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ABSTRACT 



Since 1991, the National Science Foundation has 



conducted a series of annual competitions for awards to states under 
the Statewide Systemic Initiatives (SSI) program. This document 
reports on the progress of 21 SSI initiatives through the 1992-93 
academic year. In chapter 1, "Introduction," a framework for 
assessing systemic reform is established and an overview of the first 
year evaluation activities is presented. Chapter 2, "Visions for 
Change: Scientific Literacy and Mathematical Power for All," 
discusses the visions for improved teaching and learning which are 
driving the systemic initiatives and the broader reform efforts in 
which they are embedded. Chapter 3, "Strategies for Reform," 
describes states 1 strategies for realizing their visions of change. 
Emphasis is placed on those parts of the system that are being 
targeted for reform and the theories of change underlying the states' 
approaches. Chapter 4, "The Governance of Statewide Systemic 
Initiatives: Building Collaboration, Consensus, and Coherence," 
describes the ways in which states have chosen to govern their 
systemic initiatives, raising questions about the potential strengths 
and weaknesses of each. The final chapter, "Looking Ahead: Challenges 
for Further Evaluation," concludes the report with a discussion of 
the issues involved in measuring the success of NSF's Statewide 
Systemic Initiatives program. (ZWH) 
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Foreword 



The Statewide Systemic Initiatives (SSI) Program was established in 1990 as part of 
NSF's efforts to reform the delivery of K-12 mathematics and science instruction in the 
United States. Grants of up to $2 million per year for 5 years may be awarded to states to 
bring together partnerships of institutions (State Departments of Education, school 
districts, State Agencies for Higher Education, Institutions of Higher Education, 
businesses/industry, etc.) to formulate ambitious, coordinated, coherent, and 
comprehensive approaches to statewide reform of mathematics and science education. 
NSF made 10 awards in 1991, 1 1 in 1992, and 5 in 1993. 

In 1992, the Education and Human Resources (EHR) Directorate undertook a 5- 
year evaluation of the SSI Program, which resides in NSF's Office of Systemic Reform. 
This volume is the report of descriptive findings from the first year of that evaluation. It 
presents a conceptual framework of systemic reform and establishes a baseline against 
which progress can be measured in subsequent years. Simply put, to measure the impacts 
of NSF's systemic education initiatives, EHR must invent "systemic evaluation." This 
first-year report is a vital step in the process of invention. 

The SSI evaluation is sponsored by the Division of Research, Evaluation and 
Dissemination (RED), and is being carried out by SRI International, Menlo Park, CA. 
The RED Program Officer for the SSI evaluation is Susan Gross. NSF welcomes 
comments on its programs and products. 

Daryl E. Chubin 
Division Director, 

Research, Evaluation and Dissemination 
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Executive Summary 



Since 1991, the National Science Foundation (NSF) has conducted a series of annual 
competitions for awards to states under the Statewide Systemic Initiatives (SSI) program. 
In 1991, 10 awards were made, and in 1992, an additional 1 1 awards were made. This is 
a report on the progress of these 21 SSI initiatives through the 1992-93 academic year. 1 
The report was prepared as part of an ongoing evaluation by a team of researchers from 
SRI International and its partners, the Consortium for Policy Research in Education, 
Policy Studies Associates, the Council of Chief State School Officers, and Woodside 
Research Consortium. 

This first-year report is an interim document. As such, it is intended primarily to be 
descriptive and to establish a baseline against which progress can be measured. The 
description allows different systemic reform strategies to be characterized and raises key 
analytic questions that will need to be addressed in the future. By the third-year 
evaluation report, we will begin to draw conclusions about the effectiveness of the SSI 
initiatives. 

Our evaluation is based on a conceptual framework of systemic reform, the model of 
change that NSF used to organize the solicitation for SSI awards. The underlying premise 
of the model is that attainment of world-class standards in mathematics and science 
education will require the replacement of isolated and piecemeal reform efforts (such as 
implementing a new curriculum) by ambitious, coordinated, coherent, and comprehensive 
approaches involving many different aspects of the education system, ranging from the 
preparation of teachers, to the selection of instructional materials, to the assessment of 
student learning, to the development of public support for improved mathematics and 
science education. NSF provided the states with a significant degree of latitude in 
deciding how to formulate and implement a systemic reform strategy. 

Visions for Change: Scientific Literacy and Mathematical Power for All 

The starting point for systemic reform is the creation of a set of ambitious learning 
goals for all students, including those often underrepresented in mathematics and science, 
such as minorities, females, and students living in poverty. All of the SSIs are seeking to 
develop and articulate clear goals for what students should know and be able to do. On 
the surface, these emerging visions appear remarkably uniform, especially in mathematics, 

^ive additional SSIs, funded in 1993, have not been included in the evaluation to date. 



ERLC 



ix 

a 



where the Curriculum and Evaluation Standards for School Mathematics published by 
the National Council of Teachers of Mathematics (NCTM) is highly influential. Because 
there is not yet a clear set of standards in science, there is less unanimity from state to 
state in this content area. 2 

Despite agreement on the general goals of reform, states vary considerably on the 
specifics of their visions. In particular, the states differ in how they address the issue of 
integration across and within content areas Vermont, for example, is seeking to develop 
an integrated mathematics and science curriculum, whereas most states are, at least in the 
short run, content with maintaining separate content areas. Montana provides an example 
of an attempt to integrate technology fully in teaching and learning mathematics (at least at 
the secondary level), in contrast to many other SSIs. 

States also vary in how they approach the key issue of equity Some of the SSI 
states have focused attention on particular underrepresented groups (for example, 
Connecticut, Michigan, and California concentrate funds on schools and districts sen/ing 
underrepresented students); others are still grappling with ihe best approach to equity. 
States are especially challenged to reduce "tracking" in schools. At the elementary level, 
this effort often involves a clash with the basic skills culture of compensatory education 
programs, including those funded with Chapter J dollars At the secondary level, the key 
issues revolve around vocational versus college-bound tracks both within and across 
schools. Some states have actively tackled these problems. Montana has taken the bold 
step of creating a single secondary mathematics curriculum for all students. Delaware has 
included vocational schools in its reform efforts. Most SSI states are still working out 
ways to provide high-quality mathematics and science for all students. 

States are employing two policy instruments for communicating ambitious learning 
goals throughout the education system: curriculum frameM>orks and student assessments. 
Although there is disagreement as to what a curriculum framework should be (for 
example, only some frameworks focus on how mathematics or science should be taught), 
almost all the SSIs are using curriculum frameworks (or actual curriculum materials) to 
communicate a vision of ambitious learning goals. 

Some SSI states, such as Kentucky, are relying heavily on student assessments as a 
way of providing clear messages to teachers, families, and students about what students 



2 Ncithcr the mathematics nor emerging science standards address implementation issues. Clearly, this is 
a central function of the SSIs. 



should know and be able to do. 3 Creating, adapting, and implementing high-quality 
assessment instalments that provide useful information to a range of audiences in the state 
poses formidable technical and political challenges. Some SSI states have used NSF 
funding as a means for moving ahead with new assessments, but the amount of funding 
required is typically very large. For financial, political, or other reasons, many states have 
yet to institute plans for ambitious state assessment systems. 

Strategies for Reform 

States have chosen a diverse set of strategies to achieve their visions of improved 
mathematics and science education. Some states place greater emphasis on districts and 
schools devising innovative programs; others seek to spread state-developed conceptions 
of good practice throughout the system. Some focus on the early grades; others target the 
middle grades; fewer address the problems of high school. Among the most prevalent 
reform strategies are: 

• Model or demonstration schools, where funds are provided, typically on a 
competitive basis, to a relatively small set of schools (sometimes districts) to 
develop effective strategies that will be disseminated to other schools (9 states). 

• Training of lead teachers or school teams to alter the teaching and learning of 
mathematics in their schools (14 states), 

• Development and dissemination of new cxirricula or curriculum units to replace 
what is currently taught in the schools (2 states). 

• Restructuring ofpreservice education programs to prepare a new generation of 
teachers better able to put new pedagogical practices in place (10 states). 

• Mobilization of public opinion to build public support for the goals and vision of 
the statewide systemic initiative (18 states). 

Most states use some combination of these strategies to promote their reform 
efforts. Across all states, however, there is a special focus on building the capacity of 
school-level educators to implement the reform agenda. States report spending a greater 
proportion of their funds (28%) on professional development for currently practicing 
teachers and administrators than for any other activity. In addition, a large portion of the 
SSI fonds that states pass on to local districts and schools (13% on average) supports 
professional development activities. 

3 On the other hand, some states still rely on outdated assessments that send messages about student 
learning that do not match new goals. 
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In contrast, states are putting much less emphasis on reforming the preparation of 
teachers. Only a handful of states are spending more than 10% of their funds on 
preservice education (compared to a large majority doing so for inservice). Perhaps this 
underscores the difficulty of simultaneously reforming both K-12 education and higher 
education, which in many states have separate governing boards and political systems. 
Interestingly, within the K-12 arena, there is a greater focus by SSIs on the elementary and 
middle grades than on secondary schools. 

A noteworthy contribution of the SSI program is that many states are focusing 
attention on the importance of mobilizing public support for reform of mathematics and 
science education. Public awareness and support are often underestimated by reformers, 
who may believe that "everyone" accepts improving mathematics and science education as 
high-priority goals. There is some evidence that the public does not see the same need. 
Four SSI states report using more than 10% of their NSF funds for public-awareness 
activities, and all bu> three report using some (but less than 10%) of their SSI funds for 
this purpose. 

Overall the SSI states were successful in obtaining more than a 1 00% "match " of 
other funds to use in support of their systemic initiatives. Of these other funds, the 
largest amount (about 36%) came from the state and local portion of the Eisenhower 
Mathematics and Science Education Program. Almost as much (34%) came from state 
sources. 

Finally, most SSIs are integrated into a larger series of state reforms that have been 
underway for years. In California, for example, SSI funds are used to support ongoing 
reform efforts geared to the development of curriculum frameworks, adoption of 
instructional materials that meet state standards, and piloting of a number of performance- 
based assessments. In a few cases, however, the SSI is at odds with other state reform 
efforts. Some states have had relatively few reforms under way before the SSI. In those 
states the systemic initiative has set the agenda for major reform, with the potential of 
introducing a framework for the development and delivery of improved mathematics and 
science education. The reformers in these states face the disadvantage of having relatively 
little to build on. 

Governance and Collaboration 

The ways that institutions and agencies interact and collaborate in the interests of 
promoting systemic reform are likely to have an effect on how well they succeed. To 



date, the SSI states are governing their efforts at systemic reform in very different ways. 
It is too early to predict how these governance patterns will affect the SSI initiatives. 

7he norm among the SSI states is to involve several different institutions and 
agencies from the outset Within this common pattern, two overlapping but distinct 
approaches are to organize the efforts through internal (state agency) leadership or 
through external leadership. 

External leadership for systemic reform is a potentially interesting innovation. In 
several states, whole new entities have been created to govern the SSI, such as the 
Connecticut Academy for Education in Science, Mathematics, and Technology or the 
Maine Mathematics and Science Alliance. Advantages of such an approach include 
avoidance of red tape, the ability to hire or attract better-qualified staff; stronger ties to the 
mathematics and science communities, and the ability to survive changes in state 
leadership. 

Internal leadership is somewhat more common for the SSIs than external 
leadership. Advantages are said to include closer ties with other state programs and 
reform initiatives, greater knowledge of federal programs, and an understanding of state 
policy-making mechanisms. 

Whether one form of leadership, internal or external, is better for particular states 
remains to be seen. The lessons learned about governance will yield alternative models 
that states can share and that NSF can disseminate. 

Looking Ahead: Challenges for Further Evaluation 

The initiatives supported by the SSI program are complex, and many are only in the 
early stages of implementation. As we proceed with the evaluation of the program, we 
need to address a number of challenges. 

First, SSI is not implemented in a vacuum. A wide array of other reform initiatives 
is at work in many states, ranging from school finance reform to changing the federal 
Chapter 1 program so that it encompasses more than just "basic skills." To understand the 
development of the state initiatives and to judge their contribution to reform, we must 
evaluate them within this broader context of reform. 

Second, the evaluation to date has focused principally on the infrastructure that 
SSIs are building to improve their systems of mathematics and education. Yet, the 
ultimate goal of the SSI program and the individual state initiatives is to affect the 

xiii 1 3 




teaching and learning of mathematics and science. In the future, we will focus more on 
districts, schools, teachers, and students. Understanding the interactions of different levels 
of the education system is an important goal. 

Third, certain common reform strategies, such as the development of curriculum 
frameworks and new assessment systems, require considerable in-depth analysis. These 
areas will continue to be significant parts of the evaluation each year. 

If we meet these challenges, the evaluation will continue to reap lessons from the 
states that will benefit NSF's work with other programs and initiatives (e.g., the Urban 
Systemic Initiatives, Collaboratives for Excellence in Teacher Preparation, and the Rural 
Systemic Initiatives). Finally, this knowledge can be disseminated to diverse audiences 
outside NSF. 
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1 INTRODUCTION 



In 1990, the National Science Foundation launched the Statewide Systemic 
Initiatives (SSI) program soliciting proposals from states to create broad-based statewide 
coalitions to undertake ambitious, comprehensive, and coordinated reforms of science, 
mathematics, and technology education. This solicitation marked a new approach to 
NSF's support for the improvement of mathematics and science education. The typical 
SSI award is for as much as $10 million over 5 years. This program is significant in that it 
calls for statewide, comprehensive initiatives supported by the governor and involving all 
key players in mathematics and science education. 

With SSI, NSF has redefined its role in supporting change, reflecting a new vision of 
what it will take to improve student performance in mathematics and science and meet 
world-class standards. In this new role, NSF sees itself as a "strategic broker," which 
'"brings together those who have identified problems with those who have the resources 
and skills to help solve those problems... [and] provides not only money in terms of 
resources, but serves as a catalyst for new ideas and processes" (Massey, 1993). The 
strategy is to prompt states to bring together all key players (teachers, parents, university 
researchers, business leaders, etc.), to articulate a vision of effective mathematics and 
science education in the state, and then identify the elements of their systems of education 
that need to be changed to realize this vision (Massey, 1993). 

This strategy reflects a growing consensus among education reformers that 
significant improvement in the nation's schools will require more comprehensive and 
coherent approaches, driven by clear and ambitious learning goals (Fuhrman, 1993; Smith 
& O'Day, 1991). The SSI model of change supports systemwide, coherent reform of 
curriculum frameworks, instruction, teacher education, and student assessment. These 
reform efforts are to be designed and implemented with the cooperation of all key players 
in the mathematics and science communities. This model moves beyond isolated activities 
that are typically designed to target only one component of the educational system. 

The Statewide Systemic Initiatives program represents both a change from previous 
NSF approaches to supporting reform and an exceedingly ambitious agenda for improving 
the state of mathematics and science education in the nation. NSF hopes that the 
statewide systemic initiatives will alter statewide systems of education, leading to 
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significant improvements in classroom practice consistent with both state goals and 
accepted professional st andards (e.g., National Council of Teachers of Mathematics, 
1989) and in student learning. 

NSF has contracted with SRI International and its partners, the Council of Chief 
State School Officers, the Consortium for Policy Research in Education, and Policy 
Studies Associates, to conduct an evaluation of the program. This document is the first- 
year evaluation report. Our intention here is not to provide a preliminary report card, 
either on the SSI program as a whole or on individual systemic initiatives. Rather, we use 
this report as an opportunity to describe patterns across states' approaches to systemic 
reform and then to raise key questions that will be addressed in the evaluation. 

In this introductory chapter, we first outline the framework we are using to map 
statewide systemic reform initiatives. We then describe how this framework can be used 
to examine very divergent state approaches to reform. Next, we briefly review our data 
collection activities during the first year of the evaluation, which form the basis for the 
subsequent chapters of the report. Finally, we provide an overview of the rest of the 
document. 

A Framework for Assessing Systemic Reform 

The Statewide Systemic Initiatives program is based on a general model of change, 
commonly referred to as "systemic reform," which can be used as a framework for 
examining the progress of the statewide systemic initiatives and the broader state reform 
efforts in which the projects are located. By "framework," we are referring to a 
conceptual map that identifies key factors and asserts likely relationships among them 
(Miles & Huberman, 1984). 1 As such, a framework helps to guide data collection and 
analysis. Experience in the field and initial data analysis in turn can help to refine the 
conceptual framework. A well-specified and sufficiently refined framework can ultimately 
be useful not only for guiding research but also for specifying key improvement points in 
the system for policy-makers and practitioners. We present such a framework in Figure 1, 
which we describe in greater detail below. 

The logic behind systemic reform is that changes in the teaching and learning of 
mathematics and science are needed to improve student performance to meet world-class 

1 Such a framework is analogous to but different from a statistical model, which specifies variables and 
hypothesizes the strength of directional relationships among them that can be empirically tested. 



ERLC 



2 



standards These changes must begin with a clear and ambitious vision of what students 
should learn and be able to do (Fuhrman, 1993; Smith & O'Day, 1991). Central to this 
vision is the belief that such high standards must be applied to all students, not just those 
who are preparing for advanced careers in mathematics and science. In particular, this 
vision represents a dramatic expansion of both access and attainment in learning 
mathematics and science for traditionally underrepresented groups (O'Day & Smith, 
1993). 

Creating such a vision, especially one that can affect practice, however, requires the 
unprecedented collaboration of key players in the mathematics, science, education, policy- 
making and business communities. Conflicts between key interest groups and central 
communities, such as teachers, local administrators, and state policy-makers, can make it 
impossible to build the consensus, capacity, and commitment necessary to create systemic 
reform (Elmore & McLaughlin, 1988). The composition of the leadership group that 
emerges, the nature of their participation, their commitment to the vision that is 
developed, and their willingness to sustain the effort may be important factors in the 
success of the systemic initiatives. 

This vision of ambitious learning goals, supported by the leadership and 
collaboration of the key institutional and individual players, needs to set forth coordinated 
and coherent policies and practices at both the state and local levels. These efforts are 
meant to send a clear and reinforcing message through the system in support of the 
ambitious learning goals, build support for the vision, and provide the assistance, 
flexibility, and pressure necessary for practitioners to succeed. For the sake of discussion, 
we place these policies and practices into four general categories: policy instruments, 
capacity building, authority relations, and mobilization of opinion. 

Policy instruments refers to a range of practices, inducements, and mandates that 
seek to alter what takes place in classrooms (McDonnell & Elmore, 1987). Examples of 
such instruments include policies that change requirements for teacher certification, rules 
about the adoption of materials and tools for the classroom, and curricular guidelines. For 
example, the creation of a high-stakes accountability system (as in Kentucky, where 
rewards 'or sanctions are to be meted out to schools depending upon students- 
performance over time) can be an effective policy instrument to get teachers to focus on 
reform goals. 



n 



o 
o 

o 

CO 
1— 

a 

w 

O 

E 
< 

C 

CO 



O 
i— 

<D 
O 

O 



o 
o 

QL 

w 
o 

3 
E 
< 

■a 
c 

CO 



o 
sz 
o 
o 




X 

c 
o 
o 

13 
o 
o 




L 



o 

c 
o 
O 

G) 

2 

CO 










c 




>» CO 




O - 




Authorit 
Relation 




lizat! 

0T 

Inlor 












2 




Q- 
O 

o 
-J 

O 
CO 
J3 
"D 

% 

LL 



T 



C 

Q- O 

2 -D | 
©eg 

CO 5 

$ — 
_J o 
O 




o 

LL 
UJ 
CC 

o 

UJ 

to 
>- 
c/> 

o 

< 

< 

> 

UJ 

QC 

o 

LL 
CC 

O 

UJ 

< 

DC 
LL 

< 



UJ 

CC 

o 

LL 



O 



Capacity building refers to a host of activities designed to prepare individuals and 
institutions at all levels of the system to acquire the skills and knowledge necessary to 
accomplish the ambitious tasks that systemic reform demands of them (David, 1993). 
Educators at all levels of the system face capacity problems. The biggest capacity issue 
facing the systemic reformers, of course, is the competence of school practitioners to 
teach the more challenging standards. Teachers are being asked to have greater 
understanding of their subjects, to adopt new methods, to be more accountable, to change 
their views of students, to cope with more heterogeneous classrooms, and to take a more 
active role in governance (Fuhrman, 1993). Yet, whether at the state, district or school 
level, staff typically need greater capacity to prepare teachers to use the new frameworks, 
methods, and assessments. They need capacity to provide on-site assistance to support 
teachers' efforts to use new techniques (David & Goren, 1993). 

There is also an issue of fiscal resources, especially during difficult economic 
conditions, and especially given the inequities that characterize school spending within and 
across states. Change in mathematics and science education will demand resources for 
staff time, materials, training and assistance, and technology. 

Authority relations refers to the distribution of power through the intergovernmental 
system (e.g., state education agencies [SEAs], local school districts, and building-level 
staff). Systemic reform calls for clarifying and raising standards and direction from the 
"top" (e.g., the state) combined with greater authority and accountability at the "bottom" 
(e.g., the school level). This aspect of systemic reform, often referred to as "the authority 
for accountability swap," reflects the recognition of the inability of centralized, top-down 
regulation to build practitioner commitment and release the energies of school staffs. 
Proponents of systemic reform insist that individual schools retain broad discretion over 
instruction. From this perspective, the state should provide direction and support through 
curriculum standards, instructional materials, assessments, and staff development, but 
should refrain from prescribing the details of classroom practice (Newman & Clune, 
1992). 

Mobilization of opinion refers to efforts to get professionals, e.g., teachers, higher 
education faculty, administrators, and policy-makers, to adopt a common vision of reform 
and to get support for that vision from the public, especially parents and business leaders 
(Elmore & McLaughlin, 1988). Items often overlooked in discussions of systemic reform 
include persuading: state and local policy-makers to develop more coherent policy and 
focus their attention on a common vision; state legislators to appropriate sufficient funds 
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and change necessary legislation; thousands of teachers and administrators to alter their 
practice; and students and parents to accept — even demand — changes that are a central 
part of any comprehensive and ambitious reform. 

Taken together, a coherent approach to policy development, sufficient capacity 
building, flexible authority relations, and energetic mobilization of support are meant to 
create the conditions necessary at the local level to make reform possible. These 
conditions include well-prepared teachers and administrators capable of making ambitious 
goals a reality in their schools; collaborative relationships between professionals within 
schools and between schools and districts, that allow for creative approaches to teaching 
and learning; and an overall climate, both within and outside the school walls, that 
supports ambitious reform efforts. It is only within this overall context, according to the 
logic of systemic reform, that we can expect sustained and widespread changes in practice 
at the classroom level. The changes vary to some extent from state to state, but the 
overall goals for classroom practice are usually quite similar across the states, and are 
consistent with the national education goals in mathematics and science. Systemic reform 
is intended to promote change at the classroom level based on: 

• Appropriate curricula that provide challenging content for all students, 
emphasizing the importance of thinking as well as facts and procedures. 
Curricula should be organized around majc<- principles that have been identified 
as important by experts in these fields, such as the National Council of Teachers 
of Mathematics and the American Association for the Advancement of Science. 
Many curricular units are expected to have clear ties to real-world applications 
of science, mathematics, and technology that increase students' appreciation for 
the usefulness of their schoolwork. 

• Appropriate pedagogy that promotes the development of critical thinking and 
problem-solving abilities through the active engagement of students and teachers 
in investigations that lead to shared understandings of fundamental concepts and 
principals. Learning tasks are expected to incorporate instructional technologies 
as both tools and vehicles for understanding. Increasingly, classroom tasks are 
expected tc involve students in group work as well as independent work. 

• Assessment of student learning that is well integrated and aligned with 
instruction and that provides information useful to students and teachers, as well 
as to others, such as district and state policy-makers. Assessment systems that 
focus too narrowly on partial goals, such as learning facts, or that otherwise 
distort the original vision of ambitious learning goals, are to be modified or 
replaced. 
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The ultimate goal, of course, is the realization of the ambitious learning goals that 
prompted the reform effort in the first place. These goals include not only raising 
students' achievement, but increasing participation in advanced mathematics and science 
courses, improving attitudes about mathematics and science as career goals, and 
increasing access to high-quality mathematics and science among underrepresented 
student groups in these fields. 

Top-Down Versus Bottom-Up: A False Dichotomy 

The model of systemic reform guiding our framework should not ignore lessons 
from years of implementation research suggesting that practitioners' beliefs and authority 
are every bit as vital as those of people at the state level (Clune, 1993). In the above 
discussion, we have tried to outline a framework that can be used to examine reforms that 
are initiated or controlled at different points in the intergovernmental system. For 
example, we have stressed that ambitious reforms typically involve new and challenging 
roles for everyone in the system — from state department officials to classroom teachers 
(David, 1993). Similarly, we have noted that the ambitious visions of reform must be 
shared by parents and community members as well as state legislators. Coherent policy is 
necessary at each level of the system: conflicting district policies on testing are just as 
damaging as such policies at the state level. 

At the same time, our framework, consistent with most writing on systemic reform, 
does imply a directionality of influence in key components of the system. For example, the 
framework assumes that the ambitious goals for students and the general standards 
accompanying them are derived, or at least agreed on, at the larger levels of the 
system — typically the state. Similarly, technical and financial assistance often, though 
clearly not always, originate at higher levels of the system (state versus district, district 
versus school). In contrast, within this framework, we underscore the importance of 
flexibility and some control at the school and district levels. The framework also points to 
the importance of the participation and input of individuals throughout the system in 
shaping reform efforts. 

In essence, then, the framework that is currently guiding our study does place a 
certain value on state direction of reform, but it is far from the traditional top-down 
perspective that guided the reforms of the 1970s and early 1980s (McLaughlin, 1987). 
And in fact, our initial fieldwork suggests that there are few instances of traditional top- 
down change efforts in the statewide systemic initiatives in which most major decisions are 



made by a small group at the state level, nor are there initiatives that are controlled solely 
at the local level. Rather, as we discuss later, we have witnessed an array of approaches 
to reform marked as much by the diversity of interests involved as by any clear-cut control 
at either the state or local level. For example, it appears that extra-governmental teacher 
professional organizations and informal networks of teachers, concerned primarily with 
the content-specific issues in the reform of mathematics and science education, are playing 
a central role in these initiatives. 

In sum, the purpose of this framework is to guide our data collection and our 
approach to analysis. The framework's value, then, will be tested by its usefulness in these 
tasks, especially in its ability to help us describe and distinguish among the state's various 
systemic initiatives in ways that allow us to explain differences in their development, and 
ultimately in the effects on the system of mathematics and science education. 

Overview of First-Year Evaluation Activities 

During the first year of our evaluation (1992-93), NSF had signed cooperative 
agreements with 20 states and Puerto Rico. We took a two-pronged approach to data 
collection. First, for all state initiatives, we conducted ongoing documentation of the 
progress of the initiatives through document review, and through telephone interviews 
where necessary. We also invited all state evaluators to an annual meeting and developed 
a working relationship with them through which we agreed to disseminate instruments and 
evaluation documents across the states. We assisted NSF in developing an annual 
reporting format and analyzed the results of those reports for the Foundation (Hawkins & 
Zucker, 1993). Finally, for all states we collected a set of statistical indicators on 
mathematics and science education (see also Shields, Zucker, & Hawkins, 1993). 

Second, for a subset of nine states, we conducted more in-depth case studies, 
including week-long site visits to each state. These states were selected to represent a 
variety of educational and demographic contexts, as well as varying approaches to 
supporting systemic reform (Corcoran, Goertz, & Shields, 1992). The states are: 
California, Connecticut, Delaware, Kentucky, Michigan, Montana, Texas, Vermont, and 
Virginia. In the remaining years of the evaluation, we plan to continue these case studies, 
including examination of the effects of the state initiatives on a sample of districts and 
schools in each state. We will also be adding two new case study states from among the 
five recently chosen by NSF to receive SSI awards (Arkansas, Colorado, New Jersey, 
New York, and South Carolina). 
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Organization and Purpose of the Report 

The purpose of this document is to describe the statewide systemir initiatives and to 
use this description to characterize different strategies for systemic change, to pinpoint 
areas for further data collection, and to raise key analytic questions that we will need to 
address in the second through fifth years of the evaluation. As such, we are not judging 
the value of different approaches. Our gr>a! is primarily to describe and raise questions. 

In the next chapter, we discuss the visions for improved teaching and learning 
driving the systemic initiatives and the broader reform efforts in which they are embedded. 
In the third chapter, we describe states' strategies for realizing their visions of change. We 
focus on which parts of the system are being targeted for reform and the theories of 
change underlying the states' approaches. In the fourth chapter, we describe the ways in 
which states have chosen to govern their systemic initiatives, raising questions about the 
potential strengths and weaknesses of each. We conclude with a discussion of the issues 
involved in measuring the success of NSF's SSI program. 



2 VISIONS FOR CHANGE: SCIENTIFIC LITERACY 
AND MATHEMATICAL POWER FOR ALL' 



Systemic reformers focus first on the creation of ambitious learning goals for all 
students, arguing that effective change must begin with a clear vision of the desired 
outcomes (Smith & O'Day, 1991). This vision, in turn, is meant to guide policy shifts 
throughout the educational system — ultimately sending a set of coherent messages to 
teachers and administrators about both what is expected and what needs to be improved to 
succeed. 

In this chapter, we describe the visions guiding the reform of mathematics and 
science education in the SSI states. Looking across the states, we describe their goals for 
the improvement of mathematics and science education. We describe their similarities and 
differences, paying attention to the degree to which goals are embedded in frameworks 
and assessment systems and how equity issues are addressed. Finally, we raise a number 
of questions that we will seek to answer in the subsequent years of the evaluation. 

Consistent with our overall approach to the evaluation of NSF's SSI program, we 
examine the statewide systemic initiatives and the visions driving them within the context 
of broader reforms in mathematics and science education in each state. Indeed, in many 
states, the goals of the systemic initiative are the same as those that state policy-makers 
had developed before NSF funding. California, Kentucky, and Vermont— all of which 
have ongoing, systemwide reform efforts under way— are cases in point. In these three 
states, one cannot consider the states' vision and goals separately from their all- 
encompassing reform agenda. In other states, the NSF-fiinded initiative has served as an 
impetus for the further development of goals for mathematics and science education 
reform. Montana and South Dakota provide examples here — in both instances, SSI has 
served to focus key individuals' efforts and to push along an ambitious reform agenda that 
otherwise might not have been possible. Nowhere are NSF-funded initiatives unaffected 
by the larger state reform context, although for some (e.g., Texas and Florida, where 
modifications to the SSIs are now being made), it appears that they have heretofore not 
been well integrated with other state reform efforts. Thus, in this chapter, in referring to 
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reform goals, we are genera: ' discussing the broader statewide goals. When relevant, we 
note when and how the SSI fits into and supports this broader vision. 

Visions for Change 

Across the SSI states, there is remarkable consensus — at least on the surface — about 
goals for the teaching and learning of mathematics and science. These goals can be 
summed up succinctly in the phrase "scientific literacy and mathematical power for all." 
Packed into these words are a concern for both mathematics and science, a desire to see 
students come to value and to think critically across these content areas, and the aim that 
all students be able to apply their knowledge and skills to real-world situations. 
Importantly, the goal is to involve all students, not just that small percentage of high 
achievers in the mathematics and science pipeline. Given current levels of performance in 
mathematics and science, this is a remarkably ambitious goal. Michigan's vision statement 
provides a concise rendering of these general ideas: 

(1) To prepare all elementary and secondary students to become effective 
participants as workers and citizens in the 21st century. 

(2) All students will: (a) value mathematics and science and use them to make sense 
of their environment; (b) be able to communicate and reason mathematically and 
scientifically; (c) be able to apply rational, creative and critical thinking skills 
together with mathematics and science knowledge in order to participate 
responsibly in an increasingly technological society; and (d) know that 
mathematics and science are human enterprises that affect daily life. (Michigan 
Statewide Systemic Initiative, 1993) 

These same ideas of the need for schooling to create productive and competent future 
workers and citizens are reflected in California's Mathematics Frameworks: 

All students should be expected to cope successfully with the mathematics they 
will encounter outside the classroom, including the increasingly sophisticated 
mathematics demanded in most jobs and most college courses. Also discussed 
in this Framework is the mathematics required for the exercise of competent 
citizenship in a democracy. Equally important, students should appreciate the 
beauty and fascination of mathematics and approach the mathematics they will 
encounter throughout their lives with curiosity, enjoyment, and confidence. 
(California State Department of Education, 1992) 

This general agreement on the ultimate goals of mathematics and science education 
is much more firmly grounded in the case of mathematics education. It is rare that the 
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SSIs do not refer directly to the standards documents prepared by the National Council of 
Teachers of Mathematics: either the Curriculum and Evaluation Standards for School 
Mathematics (NCTM, 1989) or the Professional Standards for Teaching Mathematics 
(NCTM, 1991), or both. In contrast, there is no set of national science standards. The 
National Research Council of the National Academy of Sciences is developing such a set, 
but it will not be completed until 1994, and there can be no assurance that the documents 
will be as popular and well accepted as the NCTM volumes. In the meantime, several 
different documents are the most frequently referenced by SSIs for science. One is the 
American Association for the Advancement of Science's Science for All Americans 
(AAAS, 1989). /mother is the Content Core published by the National Science Teachers 
Association (NSTA, 1992). At one level, there is agreement in these documents that 
science should be given more prominence in the curriculum than it has had in the past; that 
students should learn concepts in greater depth, not simply memorize long lists of terms or 
facts; and that science should involve students in a wide variety of active learning 
experiences. But at other levels, such as lists of specific concepts or topics in science to 
be taught at specified grade levels, there is not yet a national consensus to nearly the same 
degree as there is in n tat hematics. 

Whether in mathematics or in science, a national consensus will not mean that 
policy-makers, educators, mathematicians, and scientists in each state agree exactly about 
what children should learn, how they should iearn it, and what efforts are needed to put 
the necessary changes in place. For example: 

• In Montana, the vision of improved teaching and learning incorporates the use of 
technology to a much greater extent than in most other states' reform efforts. 

• In Vermont, the state has committed itself to integrating the teaching and 
learning of mathematics and science at every grade level, in contrast to most 
states' efforts that retain in some formal sense the boundaries between the 
subjects. 

• In Virginia, the state initiative's focus on mathematics and science runs counter 
to the broader state reform goals of promoting integration across all content 
areas. 

That is, beneath the general rhetoric of the goals, differences often come to light. In the 
following section, we explore some of those differences, examining the degree of 
specificity in states' visions and how these visions are reflected in two key policies: 
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curriculum frameworks and assessments. We then examine how visions differ in their 
approach to equity issues. 

What States Mean by Ambitious Learning Goals 

In a number of the states involved in SSI, policy-makers have begun to rethink the 
entire vision of K-12 education (although such efforts typically started before participation 
in SSI). Illustrations of these efforts are found in the various Common Core(s) of 
Learning adopted in a number of northeastern states (Connecticut, Vermont, Maine) and 
in Kentucky's Learning Goals. For example, the Connecticut Board of Education 
adopted five statewide goals for education in 1990, addressing the general issues of 
"motivation to learn, mastery of the basic skills, acquisition of knowledge, competence of 
life skills, and understanding society's values" (Connecticut State Board of Education, 
1991, p. 14). For each of these, the state enumerates several sub-goals. Under 
"acquisition of knowledge" are the following goals for students: 

• Learn to communicate effectively in speech and writing. 

• Listen, view, and read with understanding. 

• Acquire knowledge of and ability in mathematics. 

• Demonstrate skills necessary to locate and effectively use a variety of sources of 
information, including print materials, media, computers and other technology. 

• Demonstrate decision-making, reasoning and problem-solving skills alone and in 
groups. 

• Demonstrate good study skills and skills necessary for lifelong learning. 
(Connecticut State Board of Education, 1991, p. 14) 

Such policy statements serve as a template against which specific statewide and local 
reform initiatives can be assessed. For example, in the general statements cited above, one 
finds clear support for group problem-solving, and thus for curricula that include 
opportunities for cooperative learning. Interestingly, however, science is not mentioned 
here. 

Yet such broad policy statements can be so ambiguous that they do not provide local 
educators — especially classroom teachers — with sufficient guidance to change practice. 
Moreover, because of their generality, they need not address the tough issues related to 
specifying measurable outcomes or identifying methods for helping students to reach those 
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outcomes. For example, a common complaint heard after passage of Kentucky's 
landmark reform legislation in 1990 was that the state's ambitious learning goals were too 
ambiguous to help teachers accustomed to traditional teaching methods change their daily 
routines (Shields et al., 1993). The degree to which states have taken steps to make their 
visions more specific varies widely. Two typical policy instruments for doing so are 
curriculum frameworks and new assessment practices, as we discuss in the following 
sections. 

Making Visions Real for Practitioners: Curriculum Frameworks 

Curriculum frameworks are typically policy documents that, to some degree, provide 
advice on what should be taught. They may also provide advice on how topics should be 
taught, but this is much less frequent (Sutton, 1993). Indeed, what policy-makers mean by 
"frameworks" varies considerably across states. As Pechman and Laguarda (1993) note: 

State policy leaders use the terms "curriculum frameworks," "standards," 
"assessments," and "monitoring systems" according to specific, locally derived 
definitions... Moreover, interacting and rapidly shifting political, conceptual, 
and fiscal factors lead to regular and sometimes sudden modifications in the 
emerging systems and policies that guide them (pp. 2, 7). 

At the same time, states use curriculum frameworks very differently. In Vermont, 
frameworks are meant only to guide local curriculum development; in California, they help 
to guide state textbook adoption, which has a powerful influence on content covered by 
teachers across the state. 

At their best, frameworks provide teachers and school administrators with practical 
guidance for classroom-level, subject-specific improvements. As the concrete 
representation of the broader visions for reform, frameworks can serve as the basis for 
shifts in other policy areas (e.g., assessment and teacher preparation). As such, 
frameworks are often seen as the cornerstone of systemic reform, the "structure within 
which to organize the other important educational components" (Smith & O'Day, 1991, 
p. 248). 

Many SSI states have followed this approach, complementing broader visions of 
reform with more specific recommendations embodied in frameworks. For example, in 
Vermont, where their Common Core of Learning was adopted by the State Board of 
Education in 1993, policy-makers are developing curriculum frameworks in most subject 
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areas. Kentucky, which put together its ambitious set of Learning Goals in 1990, just 
published its first set of curriculum frameworks in spring 1993. 

California followed a somewhat different path, focusing from the outset on 
developing concrete frameworks with the goal of improving classroom teaching and 
learning. Consequently, California has been developing and revising frameworks for a 
decade now, creating documents that have served as models for many other states. The 
California frameworks provide excellent examples of concrete learning experiences that 
teachers can use. For example, at grades 2 and 3, the frameworks suggest that, in terms 
of attributes and classification: 

Students begin to classify things by two (and later three) overlapping 
attributes. For example, students may decide to overlap yarn circles (Venn 
diagram with one intersection) when they are sorting students into those 
wearing watches and those with brown eyes.... (California State Department 
of Education, 1992, p. 186) 

In terms of understanding number and numeration at the same grade levels: 

Students count, compare, order, and estimate larger quantities. As they work 
with larger quantities, they group objects and investigate place value. They use 
fractions and decimals encountered in everyday situations. The students find 
halves (and other common fractions) of wholes and of groups.... (California 
State Department of Education, 1992, p. 186) 

These examples illustrate the usefulness of frameworks for practitioners interested in 
what to do differently in their classrooms to improve mathematics and science learning. 
They also reflect the effort and substantive expertise that must go into the development of 
frameworks. Partly as a result of the capacity requirements and partly because of political 
considerations, many states do not yet have curriculum frameworks in place. In fact, only 
6 of the 21 SSI states we examined this past year have implemented frameworks in both 
mathematics and science (Table 1). 

Every SSI state, however, is moving forward with framework development — in 
many cases using SSI funds to support their efforts. Vermont, for example, is using a 
portion of its NSF funds to support the development of an integrated mathematics, 
science, and technology framework. What role the development of these frameworks will 
play in promoting systemic reform is a question that we do not yet have sufficient data to 
answer. We would expect different effects across the SSI states due to the wide variation 
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Table 1 



MATHEMATICS AND SCIENCE CURRICULUM FRAMEWORKS: 
THE STATUS OF THE SSI STATES 



State 


Mathematics 
Curriculum Framework 


Science 
Curriculum Framework 


California 


In place 


In place 


Connecticut 


In place 


In place 


Delaware 


Developing 


Developing 


Florida 


Developing 


Developing 


Georgia 


In place 


In place 


Kentucky 


Developing 


Developing 


Louisiana 


Developing 


Developing 


Maine 


Developing 


Developing 


Massachusetts 


Developing 


Developing 


Michigan 


Developing 


Developing 


Montana 


Developing 


Developing 


Nebraska 


Developing 


Developing 


New Mexico 


In place 


In place 


North Carolina 


In place 


In place 


Ohio 


Developing 


Developing 


Puerto Rico 


Developing 


Developing 


Rhode Island 


Developing 


Developing 


South Dakota 


Developing 


Developing 


Texas 


In place 


In place 


Vermont 


Developing 


Developing 


Virginia 


Developing 


Developing 



Sources: Pechman, E., and Laguarda, K., Status of New State Curriculum Frameworks, 
Standards, Assessments, and Monitoring Systems (Washington, DC: U.S. 
Department of Education, 1993); Blank, R., and Dalkilic, M., State Policies on 
Science and Mathematics Education, 1992 (Washington, DC: Council of Chief State 
School Officers, 1993); interviews with selected state policy-makers. 
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both in the form of the frameworks and in their relationship to other elements of the 
education system (such as selection of instructional materials and assessment of student 
learning). 

Making Visions Real for Practitioners: Assessment and Accountability 

A second way policy-makers translate general visions for the improvement of 
mathematics and science into useful guidance for practitioners is through assessments. 
Well-designed and well-implemented assessment systems send a clear message to teachers, 
families, and students about what students should know and be able to do. In fact, one 
disjuncture in a number of states' systemic reform efforts is the mismatch between 
ambitious learning goals and the use of outdated assessment instruments that test an 
entirely different set of skills. 

Like the development of high-quality curriculum frameworks, however, the creation 
of good and usable assessment instruments that accurately test what policy-makers want 
students to learn poses significant technical and political challenges. Many states are 
considering shifts from norm-referenced tests (that often focus too heavily on "basic 
skills") to more authentic assessments that require students to perform more complex 
tasks and construct their own responses. This pattern is reflected in Table 2, where we 
outline the current status of state assessment activities in mathematics and science. All 
SSI states sponsor testing programs in mathematics and the majority (15 of 21) sponsor 
science assessments. Less than half have alternative assessments in place in either subject 
area, but most are planning to implement some form of new testing program designed to 
measure students' deep understanding and ability to apply knowledge. Creating such 
instruments, however, raises difficult technical issues of sampling, the reliability of scoring 
procedures, and comparability. Such tests are also more costly to develop,, administer, 
and score than conventional tests. 

Kentucky is an example of a state that has placed assessment at the forefront of its 
systemic reform efforts. Here, performance-based assessment in mathematics, for 
example, was put in place in the spring of 1992 at three grade levels — a year before the 
state had published curriculum frameworks in the subject area. Kentucky also has 
embedded the new assessments in a high-stakes accountability system that establishes 
achievement targets for each school in the state and promises rewards or sanctions for 
schools reaching or not reaching those goals. 
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Comments 






KIRIS includes: 
(1) transitional 
(open-ended 
and multiple 
choice); (2) 
math portfolio; 
(3) performance 
events 




MEA is 50% 
multiple choice 
and 50% 
performance 
assessment 




Science 
Alternative 
Assessment 


Planning 


Planning 


Performance 
events in grades 
4,8,12 


No 


Yes, grades 4, 
8,11, extended 
performance, 
extended 
multiple choice 


Yes, open- 
ended questions 
in grades 4, 8, 
12 


Math 

Alternative 
Assessment 


Planning 


Planning 


Math portfolios 
in grades 4, 8, 
12 and per- 
formance 
events; the high 
school portfolio 
will cover 4 
years of work 


Yes, 

performance 
based 7th grade 
math 

assessment 


Yes, grades 4, 
8, 1 1 , extended 
performance, 
extended 
multiple choice 


Yes, open- 
ended questions 
in grades 4, 8, 
12 


Science 
Competency/ 
Proficiency Test 


No 


Yes, compe- 
tency test in 
grades 3, 5, 8; 
proficiency tet : 
in grade 11 


No 


Yes, proficiency 
test in grade 11 


No 


No 


Math 
Competency/ 
Proficiency 
Test 


Yes, criterion- 
referenced 
graduation test 
in grade 11 


Yes, compe- 
tency test in 
grades 3, 5, 8; 
proficiency test 
i in grade 11 


No 


Yes, compe- 
tency test in 
grades 3, 5, 7; 
proficiency test 
in grade 10 


No 


No 


Science 
Achievement 
Test 


No 


Yes, ITBS, 
grades 2, 4,7, 9 

i 


Yes, KIRIS in 
grades 4, 8, 12 
(KIRIS is inte- 
grated across 
disciplines 
based on 75 
learner 
outcomes) 


Yes, CAT in 
grades 4, 6, 9 


Yes, MEA, 
grades 4, 8, 11 


Yes, grades 4, 
8, 12 


Math 
Achievement 
Test 


Yes, norm- 
referenced test, 
grades 4, 7, 10 


Yes, ITBS, 
grades 2, 4, 7, 9 


Yes, the 
Kentucky 
Instructional 
Results 
Information 
System (KIRIS) 
in grades 4, 8, 
12 


Yes, CAT in 
grades 4, 6, 9 


Yes, Maine 
Educational 
Assessment 
(MEA), grades 
4, 8,11 


Yes, grades 4, 
8, 12 


State 


Florida 


Georgia 


Kentucky 


Louisiana 


Maine 


Massachusetts 



9 

ERIC 



20 



Comments 


MAEP has three 
open-ended 
questions; 
designing new 
science MAEP 




Legislature has 
mandated a new 
accountability 
system bv 1996 






Science 
Alternative 
Assessment 


No 


No 


No 


No 


Yes, perform- 
ance assess- 
ment and open- 
ended questions 
being developed 
and piloted in 
arades 3-8 


Math 
Alternative 
Assessment 


No 


No 


No 


No 


Yes, enhanced 
multiple choice 
and open-ended 
questions 
piloted in grades 
3-8 


Science 
Competency/ 
Proficiency Test 


Yes, students 
must pass 
MAEP in order 
to receive state- 
endorsed 
diploma (by 
1995-96); 


No 


No 


Yes, compe- 
tency test in 
grade 10 


Yes, end of 
course profi- 
ciency tests for 
Physical 
Science, 
Biology, 
Chemistry, and 
Physics 


Math 
Competency/ 
Proficiency 
Test 


Yes, students 
must pass 
MAEP in order 
to receive state- 
endorsed di- 
ploma (by 1995- 
96); new 10th 
grade profi- 
ciency test in 
1997 


No 


Yes, compe- 
tency test in 
<jrade 5 at dis- 
trict's option 


Yes, compe- 
tency test in 
grade 10 


Yes, compe- 
tency test in 
grades 3, 6, 8, 
10 and end of 
course profi- 
ciency tests for 
Algebra 1, 2, 
and Geometry 


Science 
Achievement 
Test 


Yes, MEAP in 
grades 5 t 8, 11 


Yes, grades 3, 
8, 11, state 
approved test at 
district's option 


Yes, 3 levels at 
district's option 


Yes, CTBS at 
grades 3, 5, 8 


Yes, grades 3, 
6,8 


Math 
Achievement 
Test 


Yes, Michigan 
Educational 
Assessment 
Program 
(MEAP) grades 
4. 7,10 


Yes, grades 3, 
8, 11, state 
approved test at 
district's option 


Yes, 3 levels at 
district's option 


Yes, CTBS at 
grades 3, 5, 8 


Yes, grades 3, 
6,8 


State 


Michigan 


Montana 


Nebraska 


New Mexico 


North Carolina 
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Comments 


Portfolio 
assessments 
are linked to 
NAEP 


Changes in 
political and 
educational 
leadership in the 
state underway 


Science 
Alternative 
Assessment 


New assess- 
ments being 
designed 


Designing 

alternative 

assessments 


Math 
Alternative 
Assessment 


Yes, for grades 
4 and 8; piloting 
high school 
portfolios 


Designing 

alternative 

assessments, 

performance 

tests, and 

project 

assessments 


Science 
Competency/ 
Proficiency Test 


No 


No 


Math 
Competency/ 
Proficiency 
Test 1 


No 


Yes, Literacy 
Passport; 
proficiency test 
in grade 6 


Science 
Achievement 
Test 


No 


Yes, TAP in 
grades 4, 8, 11 


Math 
Achievement 
Test 


Yes, portfolios 
and open-ended 
questions in 
grades 4, 8 
(matrix 
sampling) 


Yes, Iowa Test 
of Basic Skills 
in grades 4, 8, 
11 


State 


Vermont 


Virginia 
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Kentucky's is an unusual case, however, because the political climate after a well- 
publicized state supreme court case allowed for a wholesale rethinking of the system of 
public education. Within that climate and at a time of relative economic prosperity, the 
state contracted with an outside firm to develop the new assessments at a cost of many 
millions of dollars, and the legislature was willing to mandate their use. South Dakota's 
and California's experiences provide better examples of the substantive, financial, and 
political challenges to putting high-quality assessments in place. 

South Dakota traditionally is a local-control state with a relatively small state 
department of education. In this context, developing and implementing an innovative 
statewide assessment system would be quite difficult. Here the NSF-fiinded systemic 
initiative acquired a commercially developed performance-based assessment and 
implemented it in a series of pilot sites. This strategy circumvented the technical 
difficulties of developing a new instrument and the political issues of a state-mandated test 
in a local-control state — while providing a measure of student progress in schools and 
districts involved in reform activities. 

California has had a tradition of strong state involvement in K-12 education, 
including a statewide testing program in place since the early 1970s. Yet, as the state's 
conception of high-quality mathematics and science education has evolved over the past 
few years, as reflected in the frameworks for mathematics and science, the state has 
struggled to keep pace with an appropriate assessment system. The state education 
department has developed and piloted a series of new assessments in both mathematics 
and science. For example, the department developed a performance-based science 
assessment at the 5th-grade level that was piloted in thousands of schools. Yet, in large 
part because of financial difficulties in the state, the formal state testing program (now 
called the California Learning Assessment System) has not yet received formal approval to 
launch the new science assessment statewide. 

As with curriculum frameworks, questions are raised about the importance of 
assessment in systemic reform. Given the political, financial, and technical difficulties 
involved in developing and implementing new assessment systems, how far can we expect 
most states to proceed? Are performance-based assessments, which require hands-on, 
project-like activities that are holistically scored, the only viable assessment approach? To 
what extent must uniform instruments be in place statewide — what is the role of school- 
or district-developed assessment instruments? Can a new system of assessments be 
designed to serve varied purposes equally well, such as providing school- or district-level 
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accountability data to policy-makers as well as classroom-level information directly useful 
to teachers and students? 

Visions for Equity 

A central theme of systemic reformers is that ambitious learning goals need to be 
held for all students, not just the small percentage of high achievers committed to careers 
in mathematics and science (O'Day & Smith, 1993). Reformers argue that systemwide 
reform cannot be accomplished if large constituencies — the poor, urban school students, 
or women, for example — are excluded or are provided with schooling of low quality. 
More practically, reformers point out that the successful development of the nation's 
economy requires workers with fundamental numeracy, scientific literacy, and critical- 
thinking skills. 

In response to such concerns, the NSF's SSI program explicitly requested that 
participating states seek ways to ensure that their systemic initiatives addressed equity 
issues. The SSI states have responded in various ways, depending in part on the 
demographic makeup of their schooling population. In some states, the issue of educating 
groups that have been historically underrepresented in mathematics and science careers is 
unavoidable. California, where more than 50% of the students in grades K- 12 are from 
ethnic minority groups, provides the clearest example. Here, no reform initiative, 
especially in the elementary grades, can succeed without addressing the needs of limit ed- 
and non-English-proficient students. The California systemic initiative, recognizing the 
special needs of its student population, targeted its recruitment effort on the one-third of 
schools in the state with the highest enrollment of low-income students. The initiative also 
has invested in translations of its materials to ensure their availability to students in 
bilingual classrooms. Connecticut, which contains a number of economically 
impoverished urban centers, has taken a similar tack. The Connecticut SSI has targeted a 
significant portion of its funds on grants to "priority districts," all of which serve poor 
urban areas. 

In other states, the target groups are different. In both South Dakota and Montana, 
science and mathematics reformers are concerned with access for Native American 
students, who traditionally have been tracked into lower-level courses. Thus, for example, 
Montana has included Native American communities in its pilot testing of new curriculum. 
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In some of the SSI states where there are not such clearly identifiable 
underrepresented groups, there have been significant debates over the best way to ' 
approach equity issues and even about which groups to target. In Kentucky, where a 
single city is the only location with an appreciable ethnic minority population, the SSI staff 
have formed an equity working group to assess how best to define and then serve 
underrepresented groups — here there is a special concern with isolated rural populations 
and with females. In Nebraska, which also has a relatively small ethnic minority 
population, the SSI staff have debated the appropriate target group — with a particular 
interest in focusing on opportunities for girls and young women in mathematics and 
science. A similar debate has arisen in Vermont — who should be targeted in working to 
provide more equal opportunities? 

Equity at the Elementary Level: The Special Case of Chapter 1 

One issue of concern is the relationship between SSI-sponsored reform efforts at the 
elementary level and the nature and quality of instruction provided to students in 
compensatory education programs, such as the federal Chapter 1 program. Chapter 1 
itself is now a $7-billion program each year, and provides millions of students 
(predominantly in elementary schools) with supplementary mathematics instruction. 
Typically, the view of Chapter 1 mathematics is highly oriented toward "basic skills," 
which is in contrast to the goals of the SSI projects and to the recommendations of the 
mathematics education community (as embodied in the NCTM Standards). 

In one large SSI state that wants to increase the participation and achievement of 
underrepresented students, the mathematics supervisor reported that "Chapter 1 is 
becoming its own system, and they aren't playing the [SSI] game. The Chapter 1 world 
doesn't trust the reform world." At least in this state — and very likely in many 
others — the Chapter 1 administrators and teachers do not appear to be convinced that the 
"new" mathematics and science, as embodied in curriculum frameworks and materials, will 
result in improved learning for their students. Indeed, they may fear that the new 
standards and assessments may further disadvantage their students. 

This issue is much larger than the SSI program. It will involve the pending 
reauthorization of Chapter 1 by Congress — and, more fundamentally, a basic rethinking of 
the goals of compensatory education and the use of compensatory funds by millions of 
administrators, teachers, parents, and even students. It also involves the examination and 
revision of public policies that have encouraged the tracking of students by academic 
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characteristics. If some portion of those in "the Chapter 1 world" are distrustful of the 
reform goals, another group that may be uncomfortable with the premises of some of the 
SSIs as they unfold are the parents of high-achieving students, who may see advantages in 
the current system of tracking. Respondents in many of the case study states indicated 
their doubts about the elimination of tracking, in part because they thought that many 
influential and active parents supported the concept. Related to this issue are fears on the 
part of these parents that new curriculum and assessment strategies, especially at the high 
school level, may disadvantage their children in the competition for places at selective 
colleges and universities. 

Equity at the Secondary Level: Ambitious Learning Goals for All? 

As states try to improve the teaching and learning of mathematics and science at the 
secondary level, reformers run up directly against a similar tradition of tracking, both 
across and within schools. Do ambitious learning goals for all students mean that all 
secondary students must be given the opportunity to learn the same content for the same 
period of time? 

Such an approach would represent a radical departure from past practice. For 
example, how would a state committed to equity for all secondary students deal with 
vocational education? Would students in vocational schools, who generally are not 
planning to attend 4-year colleges, be expected (or at least offered the opportunity) to 
master the same content as their college-oriented peers in comprehensive high schools? 
Would such expectations be unrealistic, or, more worrisome to some, would such an 
approach lead to the watering down of opportunities and expectations for more advanced 
students? Would the complete elimination of tracking preclude the teaching of some 
advanced topics in mathematics and science? What are the implications for advanced 
placement courses? 

For the group of SSI states targeting high schools, how to confront a traditionally 
tracked system is of paramount importance. Most states have not tackled this thorny issue 
directly yet, or are giving locals freedom to address it as they see fit. Some states are 
taking a proactive stance — for example, Delaware is actively involving all of its 
countywide vocational schools directly in its nascent reform efforts. 

Montana has perhaps the most ambitious plan: to create a common high school 
mathematics curriculum for all students. The plan, directed through the state's systemic 
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initiative, is to develop four distinct levels (corresponding roughly to the 4 years of high 
school), each with a series of integrated (within mathematics) curricular modules. The 
intention is that all students in the state will have access to this one curriculum, thus 
eliminating formal tracking. Faced with the reality of unequal student interest, 
preparation, and ability, the reformers recognize that not all students will cover all four 
levels during their high school years. The goal is to ensure that all students at least make 
it through the second level— thus ensuring some common exposure to key mathematical 
ideas. To further accommodate student differences, it is also recognized that some 
students may take longer than 2 years to get through Level 2 work, so there are plans for 
a sequence as follows: Level 1, Level 1.5, Level 2. Thus, the plan seeks to provide 
equality of access, a common core of experience, and opportunities both for more 
advanced work and for slower movement through the core courses. The state's 
institutions of higher education (IHEs) have already agreed that 3 years of the new 
curriculum will satisfy college entrance requirements. 

The Montana experience points to a possible solution to some of the thorny issues, 
but it also raises a number of key questions. Will all entering freshmen be prepared for 
Level 1? Does the program set some students up for failure? At the other extreme, will 
the program provide sufficiently challenging opportunities for higher-achieving students? 
These, of course, are the bdAc questions about any effort to provide simultaneously an 
ambitious and equitable academic program. 

Summary and Questions 

Across the SSI states, we find remarkable agreement on the general vision for high- 
quality mathematics and science teaching and learning— reflecting an emerging national 
consensus in these content areas. Not surprisingly, this agreement begins to disappear 
once we move beyond general statements and examine specific policies and practices. 
Even in those states that have sought to put their visions into practice, there is much 
controversy over the best approach. Montana and South Dakota have sought to 
implement new curricula directly into pilot classrooms and schools. Kentucky and 
Connecticut have used the leverage on state-mandated assessments to communicate their 
new visions for student learning. Vermont, through a great deal of local participation, has 
looked to change its entire system of education. All states face — and few have directly 
confronted— the challenge of providing equal, high-quality, and appropriate learning 
opportunities for all their students. 
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After the first year of our evaluation, we come away with more questions than 
answers concerning the difficult issues involved in developing statewide visions of 
ambitious learning goals for all students. In particular, we are interested in: 

(1) The importance of state-developed curriculum frameworks and assessment 
systems. Are these the best tools to ensure that the vision of learning is 
adequately communicated throughout the system? 

(2) Can ambitious visions be too ambitious? What happens when the state reform 
effort cannot live up to the expectations it has set? In Kentucky, the standards 
for world class achievement are extremely high — for how long will the 
public — and their representatives in the legislature — tolerate such a high 
"failure" rate? 

(3) How can states provide equal opportunities to all students without either setting 
unreal expectations for some students or failing to challenge other students? 

(4) How important is the process through which a state's vision is developed? In 
particular, how democratic must the process be to ensure broad acceptance of 
the new vision? 



(5) How do states approach the different content areas? In particular, is science 
not considered a "basic" in the same way as mathematics? 

(6) National standards in science are several years behind the mathematics 
standards. Will the SSI states accept the new science standards, and, if so, how 
much will visions need to change to accommodate the new standards? 

(7) Will the vision of mathematical power and scientific literacy extend beyond the 
traditional academics to include vocational and technical education? 



(8) Will states' visions for change effectively include reform of higher education 
(notably including preservice teacher education) as well as reform in grades 
K-12? 



We will try to answer these questions in future years. 
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3 STRATEGIES FOR REFORM 



This chapter provides preliminary information about characteristics of the statewide 
systemic initiatives and the strategies they are using to reform mathematics and science 
education. Based on data collected primarily from state-level stakeholders during the first 
year of the evaluation study, it begins the process of describing, comparing, and assessing 
the various strategies being used by the SSIs to achieve the ambitious goals we discussed 
in the preceding chapter of the report. The data presented in this chapter were collected in 
the initial round of fieldwork conducted in the nine case study states and from the annual 
reports submitted by all the states. The chapter addresses five basic questions about the 
SSIs: 

(1) What do we know about the SSIs' approach to the reform of mathematics and 
science education, and how much do the approaches taken by the 21 states 
vary? 

(2) How do the SSIs fit into the larger pattern of education reform in the states? 

(3) From the perspective of our framework of systemic reform, what are the major 
strategies being used by the states to achieve policy coherence, build greater 
capacity for change, forge new and more productive relationships, and generate 
professional and public support for the desired changes? 

(4) Given past experience with federal and state reform efforts, what factors would 
one expect to exercise significant influence on the progress and impact of the 
SSIs? 

(5) Based on what is known and what might be predicted, what research questions 
should guide the next period of data collection? 

Empirical "answers" to the first three questions, of course, are incomplete and 
inconclusive at this point in the evaluation study. The SSIs are too new for these 
questions to be wholly answered, and the data that are available are not fine-grained 
enough to assess all the important similarities and differences in the strategies being used 
across the 21 states. However, the data do raise more specific questions to be addressed 
in the next phases of fieldwork in the evaluation study. 



9 

ERLC 



4fJ 

31 



Characteristics of the SSIs 

The SSIs are partially defined by the subject areas they address, the grade levels they 
target, the size and scope of their activity relative to the statewide system, their focus on 
equity issues, and the patterns of their resource allocations. From these basic features of 
the SSIs, a general description of the SSI program begins to emerge. These descriptive 
data also suggest potentially important similarities and differences among the 21 SSIs. 
These characteristics are described below, drawing in part on data in the annual reports 
submitted by the principal investigators to NSF. 

Disciplinary Emphasis within the SSIs 

Are there any significant differences in the amount of attention being given to 
science and mathematics across the 21 states? The data from the state reports presented 
in Table 3 suggest that science and mathematics are receiving about equal attention in the 
SSIs. To date, one state initiative (Montana) ha r been addressing almost exclusively the 
reform of mathematics, and another (Florida) is focused only on science. The other 19 
SSIs are addressing both subjects. These qualitative data about the focus of each SSI are 
consistent with fiscal data reported to NSF showing very nearly equal amounts of money 
from the Foundation being used for mathematics and for science. 

Perhaps this pattern might have been expected since NSF strongly encouraged that 
both subjects be addressed. However, the relatively balanced attention being given to 
mathematics and science is noteworthy in light of the different stages of national standards 
in mathematics and in science, which might well have led to greater emphasis on 
mathematics during the initial years of the SSIs. Nonetheless, NSF's SSI program has 
been successful in stimulating roughly equal attention to the reform of science education 
and mathematics education. 

The Levels of Education Being Addressed by the SSIs 

In contrast to the apparent balance between mathematics and science, data on the 
grade levels being targeted by the SSIs (also shown in Table 3) suggest that different 
levels of the education system are being given differential attention. The elementary and 
middle grades are the principal foci of the SSI initiatives at this time, while high school 
education is receiving less attention (although in Montana, it is the only level of the K-12 
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NC 
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NE 
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✓ 




NM 


✓ 
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✓ 




✓ 


PR 
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VA 
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✓ 
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✓ 
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system being addressed). SSI state leaders forecast that nearly 12,000 elementary schools, 
more than 5,500 junior high or middle schools, and 2,700 high schools will have been 
substantially involved in the SSIs by the end of the fifth year. Three states expect no high 
school involvement in that period, and another four report that only a small number of 
high schools will participate. 

The reasons for this lack of emphasis at the secondary level are not completely clear, 
but one can make some educated guesses. Many people believe that elementary and 
middle schools are in the greatest need of help; there is great concern about the amount of 
attention given to mathematics and (especially) science in the elementary grades and about 
the competence of elementary teachers in these subjects. Needs assessments in the SSI 
states apparently identified elementary and middle schools as priority areas for reform. On 
the positive side, most people believe that students' interests, skills, aspirations, and 
opportunities are heavily influenced by their experiences and successes in the lower grade 
levels. If no science is taught or if it is rotely taught and is boring, students will avoid it 
later on. If mathematics is poorly taught and students have difficulty with it, they come to 
feel that they are "not good at it" and avoid it later on. Changing this experience is 
especially important in raising the mathematics and science achievement of women and 
minorities. 

Another possible, far less positive reason for the greater focus on middle and 
elementary schools is that high schools and the higher education system — institutions that 
are currently receiving less attention from the SSIs — are often considered to be more 
resistant to changes in pedagogy and curriculum than elementary and middle schools. If 
this is a factor, then, possibly, changes in student expectations and competence in the 
lower grades will generate pressure for reforms in the high schools and colleges later. 

Another important factor in state designs to focus on particular grade levels is the 
limited resources available for development and improvement. This issue is discussed in 
the next section. 

Resources and Local Participation 

The Statewide Systemic Initiatives program lays out an ambitious reform agenda for 
the participating states and provides them with fiscal support that, although unusually 
large for an NSF initiative, is modest in contrast to state expenditures and the scope of the 
problems to be addressed. Consider that the expenditures on public education in 1989-90 
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for the 21 Cohort One and Two states amounted to about $91 billion (NCES, 1992), 
whereas their NSF/SSI budget in 1992-93 was $36 million, or less than .05% of their total 
education budget. This amounts to a nickel in SSI funds for each $100 of education 
expenditures in the participating states. 

However, the NSF/SSI grants do represent a significant increase in the funds 
available to the states for research, development, and improvement; furthermore, the SSI 
budgets are purely discretionary. It is also important to note that NSF anticipates that SSI 
funds will be used to leverage the allocation and targeting of other federal, state, and local 
funds (e.g., Eisenhower funds) and private funds to support reforms in mathematics and 
science education. Thus, the SSI funds provide an important and unusual resource to the 
states. Nonetheless, their size relative to state budgets and student enrollments needs to 
be kept in mind when considering the possible impact in various states and the timelines 
for going to scale with curricular and pedagogical reforms. 

The average size of the investment masks some large differences in the size of NSF' s 
investment from state to state. Both California and Montana, for example, are receiving 
$2 million annually from the SSI program. In California, this amounts to about $0.40 per 
enrolled student in grades K-12; in Montana, the corresponding figure is about $18.00. It 
is not unreasonable to expect the SSIs with substantially greater amounts of money 
relative to population to be able to accomplish their goals more quickly than SSI states 
with smaller amounts. However, NSF staff believe that larger states have greater 
resources to call upon and that federal SSI funds should mainly act as a stimulus to 
leverage these resources. 

Strategies for Going to Scale 

One consequence of these resource constraints is that many states are limiting the 
numbers of participating schools and districts. For example, Michigan and Connecticut 
have chosen to work primarily with targeted urban districts, Virginia and California have 
chosen to focus on high-poverty districts, and Kentucky and Texas are developing 
demonstration sites. At this point, primarily smaller states, such as Delaware, Montana, 
and Vermont, envision significant statewide impact on schools within the 5-year time 
frame of the SSI. 

Table 4 compares the number of districts targeted for participation in the SSIs, both 
now and in the future, with the total number of districts in the participating states. The 



state SSI principal investigators forecast that more than 65% of the districts in the 21 
states will participate directly in some SSI activity during the 5 years of funding. These 
data suggest thai the SSI program has a significant scope of action, but also that the 
efforts will have to be sustained long beyond the funding period to achieve the goals of 
universal improvement. 



Table 4 

INVOLVEMENT OF DISTRICTS 



Total Districts 
Expected to Be 

Total Districts Total Districts Involved Over 
in SSI States Involved to Date 5 Years of SSI 

Cohort 1 2,907 755 (26%) 2,125 (73%) 

Cohort 2 4,012 938 (23%) 2,400 (60%) 



N = 21. 

Source: NSF Annual Reports, 1 993, Items A1 and A2. 



However, these figures do not tell the full story of the potential impacts of SSIs. It 
is likely that in many districts the impacts will be partial: only certain grade levels, 
subjects, or schools will be affected. On the other hand, many nonparticipating districts 
may adopt or adapt the curriculum, assessment, and pedagogical reforms associated with 
SSIs, and this will increase the percentage of districts and schools affected. Many states 
expect that, as the momentum for reform builds, pressure from parents and teachers, the 
activities of professional associations, and state actions with regard to curriculum, 
assessment, teacher certification, and other policies are likely to generate pressures for 
"voluntary" local adoption of the reforms. 

Typically, after initial planning is complete, states are beginning their SSIs on a small 
scale with the expectation that they will grow larger over time. The plans for this growth 
vary from one state to another. For example, the main mechanism in Montana is a single, 
large curriculum development project that has begun to be implemented in pilot schools 
and classrooms, and ultimately is expected to reach many or all of the high schools in the 
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state (and perhaps some out of state, as well). By contrast, in states such as Connecticut, 
Michigan, and South Dakota, the major mechanisms for local diffusion will be a 
succession of grant competitions open to districts within the state; as more awards are 
made over time, the SSI initiative grows. The distribution of these different "models" or 
mechanisms of diffusion is provided in Table 5. 

State policy initiatives consistent with the SSI could also dramatically enhance its 
impact. For example, if a state changes teacher certification or recertification 
requirements or its assessment program, then every district in the state would be affected, 
although all the districts would not necessarily be "directly involved" in the SSI. Or, if the 
SSI initiative helps states in developing new alliances among varied institutions and 
organizations, these accomplishments might extend the impact of the initiative. 
Nonetheless, policy-makers must realize that the SSI program is not likely to touch every 
student in the participating states, or even directly reach every student's teacher. The 
available funds simply will not permit this level of impact. 

Allocations of SSI Funds 

Table 6 displays how the NSF SSI funds have been allocated across major domains 
of activity during the past year. The table shows that nearly 30% of the total SSI budget 
was allocated to inservice education, which constituted the single largest cai ^ory of 
spending last year. What may be somewhat surprising for an initiative that is often 
characterized as "standards-driven" reform is that only 4% of the total was allocated to 
framework development and only 6% to assessment (which is generally too expensive for 
SSI to handle the costs). Part of the explanation may be that many of the SSI states 
already have frameworks or are using state (or other federal) funds to develop them. 
Similarly, states themselves may be paying for the development of new performance 
assessment systems. Another part of the explanation is that much of the work going into 
framework development is being volunteered by teachers and faculty or being paid for by 
their institutions. 
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Table 6 

PERCENTAGE OF NSF SSI EXPENDITURES BY PURPOSE 



Category Amount 

Frameworks 4% 
Curriculum/science 4 
Curriculum/math 6 
Assessment/science 3 
Assessment/math 3 

tnservice/science 1 5 

Inservice/math 13 
Preservice/science 3 
Preservice/math 2 
Public awareness 5 
Steering committee 4 

Administration/coordination 16 
Evaluation 4 

Grants to locals* 13 
Other 6 



N = 21. 

* These funds were also used for inservice, curriculum development, assessment, and similar 
purposes, but at the district or school level. 

Source: NSF Annual Reports, 1993, Item B2. 



The NSF funds are not the only ones being used to support the SSI initiatives. 
Table 7 shows that the states were successful in finding more than a 100% "match" in 
other monies for the approximately $36 million provided by NSF. An additional $57.4 
million above and beyond NSF funds was raised by the states, according to the figures 
they provided to NSF. Table 7 also shows that, whereas the SSIs are having success 
leveraging other state and federal funds, they have had little success raising private funds. 
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Table 7 



FUNDS FOR SSI FROM SOURCES OTHER THAN NSF 



Source 


Amount 


Percent 
of Total 


Eisenhower/District funds 


$14,168,646 


24.7% 


Eisenhower/Higher education 


6,712,256 


11.7 


Eisenhower national curriculum grant 


556,710 


1.0 


Other U.S. Dept. of Ed. funds 


3,313.839 


5.8 


Other NSF grants 


5,067,616 


8.8 


Funds from other federal agencies 


62,842 


0.1 


State funds 


19,424,829 


33.8 


Private foundation grants 


794,080 


1.4 


Other 


7.307,531 


12.7 


Total 


$57,408,349 


100.0% 



N = 21. 

Source: NSF Annual Reports, 1993, Item C2. 



Integration with Other Reform Initiatives 

The recent history of the 21 states participating in the SSI program is remarkably 
varied in terms of the extent and the nature of school reform efforts under way before the 
initiation of SSI. In a period marked by a high level of state reform activity, some states 
have been more aggressive than others in pursuing statewide reforms in general, and in 
science and mathematics education in particular. To understand the implementation and 
the impact of the SSI program, it is important to keep this "bigger picture" of education 
reform in mind. 

If states were placed on a continuum representing the scope of their reforms apart 
from the SSI, California, Kentucky, and Vermont would be among those states near the 
most active end, with many other reform activities under way. States like South Dakota 
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and Louisiana would be closer to the other end of the continuum, having supported 
relatively little else in the way of widespread education reform, while a number of states, 
including Delaware, would fall somewhere in the middle. Examining the impacts of SSI in 
these varied contexts requires some understanding of the states' histories. 

In the states with a lengthy history of reform, SSI is typically an important piece of a 
broader state school reform strategy, but it is only a piece. SSI was not the catalyst for 
reform in these states, nor did it determine the direction of change, even in science and 
mathematics education. The challenge is not simply to stimulate change in mathematics 
and science education, but to integrate those changes with other state and local reforms. 
Several examples illustrate this situation. 

• Kentucky. Kentucky's SSI, called the Partnership for Reform Initiatives in 
Science and Mathematics (PRISM), is highly integrated with a comprehensive 
education reform initiative that resulted from a state supreme court decision in a 
1990 school finance case. The Kentucky Education Reform Act (KERA), which 
was the General Assembly's response to the court decision, set forth a bold plan 
for restructuring the public education system in all disciplines. KERA provides 
both an opportunity and a challenge for the SSI leadership. The opportunity is 
created by the new statewide curricular frameworks required by KERA and the 
implementation of high-stakes assessments that include both mathematics and 
science. However, the greatest challenge facing PRISM is to persuade teachers 
to change their pedagogical approach in these fields (e.g., to use calculators and 
computers more extensively) at the same time they are being asked to cope with 
KERA's changes in standards, curricula, school organization, and the results of 
high-stakes testing. 

• California. Since the early 1980s, the State Department of Education has been 
far ahead of most states in rethinking mathematics and science education, and in 
devising various initiatives to support reform in these subject areas. 
Unquestionably, the leadership group in the state had a vision of systemic reform 
long before it was fashionable. However, some pieces of the reform puzzle 
(curriculum frameworks, for example) have been much more fully developed 
than others (e.g., the state assessment system). Because California was engaged 
in "systemic" reform before the SSI award was made, the SSI funds have been 
used to fill gaps not adequately funded from other sources. Thus, although the 
SSI in California focuses on professional development in support of elementary 
science and middle school mathematics, it would be wrong to conclude that the 
state has ignored the development of frameworks or assessment or neglected 
elementary mathematics. 

• Vermont. Vermont's policy-makers are committed to a systemic approach to 
reform and to the restructuring of the public schools. They believe that Vermont 
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is the perfect "test laboratory" because of its small size. They understand 
systemic reform as a set of linked changes based on a common vision of public 
education that is consensually developed. These changes are voluntarily 
undertaken by local schools and supported by the state. The key elements are 
the Common Core of Learning, which defines the vision, and a pervasive 
instructional guidance system of curriculum frameworks and authentic student 
assessment, professional development, school restructuring, schoc '-community 
partnerships, service integration, and expanded technical education. The SSI is 
pursuing a strategy consistent with Vermont's traditions and with th<5 broader 
reform initiative. It is assisting with development of the instructional guidance- 
mechanisms and preparing school and community teams as the catalysts for 
grass-roots change. 

In cases such as these, simply tracking the expenditure of SSI funds, or narrowly 
examining the impact of SSI activities, can be misleading because the SSIs are 
intentionally embedded in larger initiatives. The data must be understood in this larger 
context, which is both more comprehensive and more complex. The vision, energy, 
authority, and funds for reform come from different sources than in other SSIs; there is a 
history to the initiatives; and effects on schools and classrooms cannot be easily attributed 
to any single program or policy. 

At the other end of the continuum of reform efforts are states like South Dakota and 
Louisiana, where the SSIs have served as catalysts for reform. This is not to say that there 
have been no reforms in these states since the early 1980s. However, the reform efforts 
have been less systemic and less coherent than those initiated in Kentucky, California, and 
Vermont. In states in which the SSI serves as the catalyst for systemic reform, its effects 
may loom larger and may be easier to isolate and track. At the same time, however, it is 
not by any means assured that the SSI will be sufficient in size and scope in these states to 
stimulate and sustain broader-scale reforms. Beginning mathematics and science 
education reform without a strong foundation of prior efforts to build on may be a serious 
handicap. 

Some states, which can be placed in the middle of the continuum, currently have a 
much larger set of reforms under way, but these are of recent origin. Delaware is such a 
state. A reform effort is under way there that is as broadly based as that in, say, 
California; however, this effort postdates the SSI award. The leadership in Delaware has 
pulled together a variety of preexisting reform strategies under a single reform agenda 
called New Directions for Education in Delaware. The SSI will serve as the development 
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arm of this effort, supporting the implementation of new strategies for curriculum and 
instruction in mathematics and science at the local level. 

In a few states, there is a question whether the SSI is, in some real sense, standing 
apart from or even potentially in conflict with other reform efforts. There is some 
evidence that this is the case, for example, in Virginia and in North Carolina. In Virginia, 
the state has recently adopted a view of curriculum and instruction that minimizes 
emphasis on the subject disciplines; mathematics and science are no longer departments or 
organizing principles in the state education agency. This approach leads to concerns about 
how well the SSI fits with the state's overall reform strategy. In North Carolina, the 
potential clash is of a different kind, based on the fact that a new regional structure has 
been created to support the SSI even though the state in the past supported at least two 
regional networks. Forming these several networks into a cohesive whole is proving 
difficult. 

Federal policy-makers need to realize that isolating the impacts of the program in the 
larger context of reform may be difficult. The program does not exist in a vacuum and in 
some instances may be supporting only particular facets of a broader state reform effort. 
It needs to be understood in a context that involves a wide variety of reform efforts, not 
only at the state level but at the federal level as well. It may turn out, for example, that the 
SSI is effective in some states precisely because it has been well timed to coincide with a 
variety of other reform efforts, or, alternatively, it may be that the SSI has its greatest 
impact in states where it serves to stimulate broader reforms that were previously absent. 

Reform Strategies in the SSI States 

Policy Coherence and Policy Choices 

Much has been written about the importance of policy coherence; it is one of the 
core concepts in systemic reform. Smith and O'Day's (1991) vision of systemic reform 
pairs ambitious, coordinated state policies with restructured governance. The logic of 
systemic reform is that coordination of key state policies will provide more coherent 
guidance to practitioners, and have more powerful effects on instruction. While states 
differ in the range of policies they attempt to coordinate, the generally accepted view is 
that the minimum set of essential policies includes state goals or vision, curriculum 
frameworks, assessment, and teacher preparation (Fuhrman & Massell, 1992, p. 4). Some 
of the SSI states, such as California, Kentucky, and Vermont, have worked hard to create 
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coherent policies that send consistent signals to the field and have a reinforcing, m jltiplier 
effect on reform initiatives. Others, such as Michigan, are engaged in comprehensive 
reviews of policies to determine how well they fit together. In many of the SSIs, the 
collaboration among many interests (including the state education agency, higher 
education, the governor's office, the legislature, education interest groups, business, and 
civic organizations) focuses attention on issues of policy coordination and creates 
pressures for greater coherence. Whether formal or informal, newly created or long- 
standing, the collaborations among state-level agencies and institutions may play critical 
roles in creating a policy environment that supports systemic reform. 

Some states, such as California and Kentucky, already have tightly coordinated, 
centralized policies and are using the SSIs to move their initiatives forward. Others, such 
as Vermont and Connecticut, are highly coordinated but, consistent with their political 
traditions, are less prescriptive and less tightly coupled. Some, such as Delaware, Texas, 
and Virginia, are moving toward development of more coherent policies and integration of 
the SSIs, although the movement is not always smooth and consensual. Still others, like 
Michigan and Montana, have not developed a statewide approach based on a coherent set 
of policies, and the SSIs there may serve as the catalyst for a more systemic approach. 

Some quantitative information about the main focus of the 21 SSIs is available from 
the survey of SSI principal investigators conducted by NSF. Table 8 shows the number of 
states spending 10% or more of their SSI budget during the past year on selected 
components of systemic reform. Using this criterion, the most common activity supported 
by SSIs was teacher professional development, with curriculum development the next 
most common. It should be noted, however, that money is not necessarily the only metric 
worth using; for example, it might require little SSI money to change a state's teacher 
certification requirements, yet the impacts might be great. 

Much has been written about the need for coherence, but less attention has been 
given to the choice of policy instruments to lead reform and to the examination of which 
instruments are likely to be most effective in bringing about the changes in pedagogy and 
curriculum necessary to meet the challenging standards envisioned in systemic reform. 
McDonnell and Elmore (1987) have suggested that a limited set of policy instruments are 
available to promote reform: mandates, inducements, capacity building, and system 
changing. A fifth category, mobilization of public opinion, probably should be added to 
these four. McDonnell and Elmore argue that the choice of policy instruments affects the 
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Table 8 

MAIN FOCI OF THE SSI INITIATIVES* 



State 


Teacher 
Inservice 


Curriculum 
DevelODment 


Frameworks 

& Standards 


Student 


Teacher 


Public 
Awareness 


Awards to 
LEAs 


CA 


✓ 










«/ 




CT 


✓ 


✓ 












DE 


✓ 


✓ 










(/ 


FL 


✓ 














GA 


✓ 








✓ 






KY 


✓ 


✓ 






✓ 






LA 


✓ 














MA 










✓ 






ME 


✓ 












✓ 


Ml 












✓ 


✓ 


MT 


✓ 


✓ 












NC 


✓ 














NE 


✓ 


✓ 








✓ 




NM 


✓ 












✓ 


OH 


✓ 


✓ 












PR 


✓ 


✓ 










✓ 


Rl 














✓ 


SD 




✓ 




✓ 






✓ 


TX 


✓ 


✓ 




✓ 








VA 


✓ 






✓ 


✓ 






VT 


✓ 


✓ 




✓ 


✓ 




✓ 



* ✓ Means the state spent 10% or more of the SSI funds on this category during the past 
"LEAs = local education agencies. 

Source: NSF Annual Report, 1993, Item B2. 



45 



64 



impact of reform initiatives, and that more needs to be known about how these different 
instruments work, given variation in purposes, contexts, local capacities, etc. 

In only a handful of SSI states — California and Texas (instructional materials), 
Kentucky and Puerto Rico (frameworks), and possibly in the future, Delaware (standards) 
and Kentucky and South Dakota (performance testing) — do state mandates appear to be 
playing significant roles in the implementation of the SSIs. And, with the possible 
exceptions of South Dakota and Delaware, the use of mandates in these cases does not 
represent any change in traditional state-local relationships. 

Inducements of various types (grants and publicity, to name two) together with 
capacity-building activities (such as professional development) are the preferred tools of 
implementation. All the SSIs are using inducements in some form to persuade local 
policy-makers and teachers to undertake reforms in curriculum and pedagogy. In a 
subsequent report, we expect to discuss these various inducements in greater detail. 
Capacity building and systems changing are so important to the SSIs that they are 
discussed separately below. 

Building Capacity 

Teachers and administrators need to believe that the reforms advocated in the SSIs 
are possible and that they are capable of implementing them. They must believe that the 
intellectual, time, fiscal, and political constraints to the changes in teaching and curriculum 
can be overcome. New forms of support will be needed. If every teacher must become 
more expert in teaching mathematics and science, then an extraordinary professional 
development effort will be needed. If teachers are expected to teach "hands-on" 
mathematics and science, then good materials must be accessible, and they must be 
replenished. School resources will need enrichment or reallocation to support more 
laboratory work and field trips. 

Undoubtedly the most common focus of the SSIs is mobilizing professional 
networks and improving the capacity of the education system to deliver high-quality 
instruction in mathematics and science. This includes such activities as providing teachers 
with professional development services (inservice), developing and implementing new 
curriculum materials, and building networks of model or pilot schools and programs. 



46 85 



Demonstration Sites and Pilots. One strategy for building capacity is the 
development of demonstration sites. Georgia, Kentucky, Maine, Massachusetts, 
Michigan, Montana, Puerto Rico, South Dakota, and Texas are using this strategy. 
Demonstrations can provide solid evidence that reforms are feasible, and provide 
opportunities for observation and hands-on learning for teachers. Given adequate 
resources, they can serve as effective regional training institutions. They provide visible 
examples of progress for the public and can help generate public demand for reform. 
However, they also can inhibit reform if they are perceived as being given excessive funds 
or special treatment by the state. Demonstration sites have been often used as reform 
strategies, but have not always proved very effective because reforms tend not to spread 
to other sites. They may be much more powerful when combined with the other policy 
instruments employed in a systemic initiative. 

Teachers as Trainers. Turnkey training is another popular strategy for 
dissemination of new practices. A number of states are using turnkey trainers or lead 
teachers. Kentucky is preparing cadres of teachers to serve as trainers for elementary and' 
middle school mathematics and science and for technology. Virginia is training lead 
teachers, two from each participating school, to introduce reforms to their peers. They 
also provide training for school administrators. California also trains pairs of teachers 
from participating schools, but supports them with a network of consultants. Vermont is 
offering training to school teams, including principals. 

The success of these lead teacher or teacher-trainer models may depend not only on 
the quality of the training but on the amount of time they have to work with their peers, 
the opportunities for continued interaction with other colleagues who experienced the 
training, the continuation of the training through booster sessions, , the critical mass of 
committed and trained teachers in a school or a district, and the availability of materials. 
Some of the states are addressing some of these conditions; few appear to be addressing 
them all. Above and beyond qualified trainers, school and district leadership will be 
needed and must be encouraged by the SSIs. 

Professional Development. All the SSI states are making heavy commitments to 
professional development. Some are emphasizing lead teachers or school teams, as 
described above. Others are offering programs for individuals. This focus on professional 
development reflects two general findings from the research on reform. First, many 
teachers, especially elementary and middle school teachers, lack the subject-matter 
knowledge needed to teach to the new standards (Shields et al., 1993). Second, existing 
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structures for professional development may not be adequate to the task of providing 
teachers with the needed assistance. Not only is the current system fragmented, 
underfunded, and poor in quality, but the very approach taken— training programs 
intended to transmit knowledge — does not fit the demands of current reforms (Little, 
1993). In response to these shortcomings, new models of professional development are 
emerging, such as subject-area collaboratives and teacher networks, professional 
development schools, and partnerships between schools and colleges that recognize the 
expertise of teachers and provide opportunities for dialogue, exploration, and critique 
among teachers and between teachers and others involved in reforms. 

Some of the SSIs are experimenting with similarly creative approaches. Vermont 
has developed regional networks of teachers to implement its portfolio assessment, and is 
using the scoring of the portfolios as a major opportunity for professional development. 
Delaware is working with school teams who are leading school restructuring efforts. 
Professional associations are playing important roles in Connecticut, Montana, and 
Nebraska. Ohio is pairing teachers with scientists from the business sector. Many states 
are running summer institutes for teachers that bring together content experts and teachers 
in a collegial setting. These new approaches bear close monitoring to determine whether 
they are more cost-effective than the workshop model so commonly used by states and 
school districts. 

Mobilization of Support 

One of the major obstacles to curriculum reform is public acceptance of the need for 
change and support for the envisioned reforms. Although there is general dissatisfaction 
with public education, parents across the nation remain satisfied with the schools that their 
children attend (Elam, Rose, & Gallup, 1992). Gallup Poll results also show that fewer 
than a quarter of Americans are even aware that there is a national goal for American 
students to become first in the world in mathematics and science achievement by the year 
2000 (Elam, Rose, & Gallup, 1992). Moreover, when asked to rate the importance of the 
national education goals, they do not rate this one especially high (Elam, Rose, & Gallup, 
1991). There is some evidence that the public simply does not see the same need for 
higher academic standards and more rigorous curriculum that the policy community and 
business leaders see, but feels instead that students need to master the basic skills and learn 
more discipline to be competitive in the workplace (Public Agenda Report, 1990). The 
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importance of public awareness and support i& often underestimated by reformers, who 
may believe that "everyone" accepts the importance of mathematics and science education. 

Added to this general problem of public lack of understanding or lack of support for 
high standards are the debates over "values" in the curriculum being raised by religious 
leaders and conservatives. This grass-rocts opposition to reforms can be a particular 
problem for science education in a number of states. Finally, there is the underlying 
problem of who pays for reforms. In many states, the voter backlash against higher state 
and local taxes remains a powerful force. As the number of households with school 
children declines, the willingness of the electorate to support school reforms grows more 
problematic. 

Recognition of the need for building public support for sustained reform has given 
rise to one of the unique aspects of the SSIs: public relations campaigns on behalf of the 
reform of mathematics and science education. Many of the SSIs are developing and 
implementing such campaigns. This focus is a potentially important contribution of the 
SSI program as a whole. Few if any other federal programs for school reform have 
mobilized such a wide variety of institutions (e.g., public and commercial television 
stations and newspapers) to focus on public awareness of the need to improve science and 
mathematics education. 

Four states (California, Kentucky, Michigan, and Nebraska) report allocating more 
than 10% of their funds to public awareness. One of our case study states, Connecticut, 
allocated about 8% but has persuaded public relations firms, newspapers, and television 
stations to make in-kind contributions and lower their fees to mount an ambitious media 
campaign. This may be essential to their success because, as one observer in Connecticut 
noted, "there is little public demand for school reform ... unless it would be to reduce the 
costs of the schools or to lower teachers' salaries." The findings in this high-tech state are 
more stark than in many others, but taken together with data such as cited above, it is 
clear that continuing emphasis on public awareness and support will be vital. The states 
are still looking for the best vehicles to reach the public; California had planned a large 
statewide campaign, but on the advice of its program staff has reconsidered focusing its 
efforts on school-level publicity. 
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Changing Roles and Relationships 

Smith and O'Day (1991) and other advocates of systemic reform argue for a 
combination of state instructional leadership and guidance and school-level discretion and 
empowerment. Curriculum control policies are the easiest policies to implement. The 
instruments — frameworks, state assessments, materials, and professional 
development — can be coordinated by the state and put into place. Finding the balance 
between these forms of state guidance and empowerment at the school level is more 
difficult. State policy-makers usually prefer control strategies because they seem cheaper, 
quicker, and more certain. The experience with the basic skills movement suggests that 
they can be effective, but they may not work as well in areas where teachers are less sure 
of their technical competence. Advocates of empowerment strategies emphasize the need 
for commitment to ensure local implementation. Empowerment strategies usually assume 
that teachers will develop curriculum, and that they will make the right choices. In 
general, the SSI states have not addressed the questions about authority relationships 
raised by Smith and O'L'-.y. Kentucky is the only &S* state that has radically altered 
school governance and moved responsibility for curriculum to the school level. However, 
the stakes are high and it is not yet clear that Kentucky's approach will work well. 
Ironically, it also has one of the most powerful and most prescriptive instructional 
guidance systems. Delaware is also engaged in school restructuring and is moving in the 
direction of mor? school-level decision-making. 

Teachers are being asked to play new roles in a number of the states, although these 
efforts fall short of comprehensive restructuring. Virginia is relying heavily on a lead 
teacher strategy, training two teachers in each school to help their peers implement the 
reforms. California is using a similar strategy. Kentucky is following a turnkey training 
model, preparing cadres of teachers to support reforms in mathematics, science, and 
technology. Vermont is training and supporting networks of teachers who work together 
to implement and score the mathematics portfolios. Puerto Rico is trying to promote 
more independence among teachers in developing their own curriculum materials. 

Some SSI states are forging new relationships between the colleges and the schools. 
Connecticut is giving grants to create dialogues between faculty and teachers, encourage 
co-teaching, and support partnerships for inservice. Kentucky is building its technical 
assistance structure around eight regional universities. On the whole, however, the reform 
of higher education (including teacher preparation) is receiving only modest attention 
under the SSI. 
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Lessons from Previous Experience with School Reform 

There is a rich literature on implementation, innovation, and planned change in 
education. On the basis of these studies, Fuhrman and her colleagues made a set of 
forecasts about the factors that would affect the outcomes of state reforms in the 1990s 
(Fuhrman, Clune, & Elmore, 1991). Based on their work and other recent reviews of the 
implementation literature, a similar set of predictions might be made about the anticipated 
course of federally funded state reform initiatives (McLaughlin, 1991; Marsh & Odden, 
1991). These predictions might go as follows: 

• Variations in state context will produce wide variation in the choice of policy 
instruments and strategies; states will focus on those aspects of the SSI agenda 
that best fit their goals, political climate, and capacity. 

• The most ambitious efforts will have the most effect. 

• Success in changing pedagogy and curriculum will depend heavily on the extent 
to which teachers are competent to make the desired changes. Acceptance by 
teachers and implementation will be enhanced by high-quality preparation, 
interaction with other teachers, and external assistance. 

• Active commitment of district and site leadership will be critical to successful 
implementation and institutionalization. 

• Implementation success will be enhanced to the extent that the reforms delivered 
clear, coherent signals to local districts and schools. 

• The implementation process will be enhanced to the extent that local actors were 
involved in shaping the reforms, planning the strategies, and assisting with 
development. 

• High-quality materials will enhance the possibilities of successful implementation 
of new practices. 

These earlier findings suggest a number of hypotheses about the SSIs that should be 
considered as the evaluation study continues to examine local implementation. However, 
these propositions may not hold true to the same degree, or they may not turn out to be 
the most important factors shaping the outcomes of the SSIs. The SSIs represent a 
departure from previous state and federal attempts to improve public education. They are 
not narrow efforts to develop new curriculum; nor do they focus exclusively on a 
particular student population. Unlike most previous programmatic efforts to improve the 
schools, they seek the mobilization of political, fiscal, and human capacity in the states on 
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behalf of fundamental and broad-based changes in teaching and learning for all students 
with regard to science and mathematics education. Therefore, the lessons drawn from 
past studies of state and federal reform efforts offer only limited guidance for anticipating 
the unfolding of these new initiatives. As a consequence, it is important to accurately 
document and describe these initiatives in the states, and not permit predictions drawn 
from previous research to narrow the data collection. 

We sum up this chapter with a series of questions we will want to address as our 
study continues during the next 4 years: 

• There are a variety of alternative strategies for approaching systemic reform. 
How do they differ? Why do states select them? Under what conditions do they 
work best? 

• States have made a wide variety of choices in balancing the breadth of their SSI 
initiatives (i.e., a larger number of grades and subjects) with greater depth. What 
are the relative payoffs of these different approaches? 

• States enter the SSI program with very different histories of implementing 
education reform. What is the difference in "value added" by SSI in states that 
already have extensive reforms under way compared with those that have few? 

• What balance do the states achieve between top-down, prescriptive strategies, 
and bottom-up, empowering strategies? What factors affect this balance? How 
does the relative mix of these strategies affect the pace of diffusion and the 
impact of the SSI? 

• What factors contribute to the development of local policy coherence and 
support for the reforms in teaching and curriculum envisioned by the SSI? 

• Is school restructuring necessary to achieve the curricular and pedagogical goals 
of the SSI? 

• What are the interactions between the SSI and a variety of other unfolding 
reform efforts, such as the reauthorization and possible redesign of the Chapter 1 
program? 
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4 THE GOVERNANCE OF STATEWIDE SYSTEMIC INITIATIVES: 
BUILDING COLLABORATION, CONSENSUS, AND COHERENCE 



Advocates of systemic reform call for more challenging standards for all students, 
significant changes in curriculum and pedagogy, new assessments of performance, and 
revisions in teacher preparation and certification, among other fundamental changes to the 
educational system. Such a vision requires, at a minimum, cottsensus on standards among 
policy-makers, practitioners, and the public; more coordinated and coherent policies 
across agencies and across levels of government; the collaboration of public, private, and 
professional organizations; and the cooperation and commitment of state and local policy- 
• makers and practitioners to achieve the necessary changes in schools. NSF's Statewide 
Systemic Initiatives program explicitly recognizes these challenges and calls directly for 
states to form effective partnerships among the key players in the mathematics and science 
communities — in government, universities, and the K-12 system. The logic here is that 
systemic reform's demands for consensus and coherence are likely to require new 
partnerships and relationships throughout the educational system. 

How statewide systemic initiatives choose to organize and govern themselves may 
determine, in part, how well they can meet these challenges. The nature of the governance 
structures of the SSIs might affect their capacity to develop consensus, achieve coherence, 
convince various groups and organizations to collaborate, and win the cooperation and 
commitment of stakeholders at ail levels of the education system. In short, an effective 
governance structure may enable the initiative to become truly systemic, to be fully 
implemented, and to persist in spite of changing political leadership or priorities. A weak 
or ineffective governance structure may doom an SSI to the same small impact and short 
life cycle as those of many other improvement projects. 

In this section of the report, we look at the structure and process that systemic 
initiative staff use to formulate policy, allocate resources, and set the goals, priorities, and 
strategies for the SSI. We examine who makes decisions, the processes used to make 
decisions, and who is accountable to whom for carrying them out. Here we address only 
governance issues at the state level, where we focused our data collection during the 
evaluation's first year. In subsequent years, we will look more closely at the organization 
and governance of reform efforts at the local level. 
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An Overview of the Governance of Statewide Systemic Initiatives 

The governance structures of the 21 SSIs vary considerably. Some of the initiatives 
are highly centralized and structured; others are only loose collaboratives. Some are 
lodged in existing state agencies; others have their locus of control outside the state 
bureaucracy. Teachers and discipline-based professional organizations are taking 
leadership roles in some SSIs but are hardly visible in others. Although all of the SSIs 
have involved representatives from appropriate state agencies, the schools, the colleges, 
and professional groups (in part because it was a condition of funding by NSF), some have 
been more aggressive than others in their efforts to involve leaders from business, the 
science and mathematics communities, civic organizations, and the media. Moreover, 
some SSIs appear to be run by their lead agency, giving only advisory roles to 
stakeholders from outside a core leadership group, while others are participatory 
collaboratives with broad involvement in decision-making. 

A few brief examples will illustrate some of the variety and complexity of SSI 
governance structures: 

Ohio Project Discovery is co-sponsored by the Ohio Boa^d of Regents (OBR) 

and the Ohio Department of Education. The project directors represent 
those two agencies, but there are also two principal investigators located 
at Miami University and Ohio State University. The governance 
structure has three tiers. First, there is a Directors' Council that sets 
policy for Project Discovery. It is composed of the project directors, 
principal investigators, statewide coordinators, and staff representing 
participating universities. Second, there is a Coordinating Council of 30 
members representing diverse institutions, which is charged with 
integrating the components of the SSI and establishing Action Groups to 
carry out its policies. Third, there are eight Regional Councils, 
responsible for overseeing day-by-day project activities within the 
Discovery regions. 

South Dakota The South Dakota SSI is overseen by a 15-member Board of Directors 
appointed by the governor, which includes representatives from 
education, business, higher education, state government, and the general 
public. The Board advises the SD SSI staff, who, in consultation with 
the Department of Education and Cultural Affairs and the state Board of 
Regents, sets policies, goals, timelines, and budgets. There is a separate 
South Dakota SSI National Advisory Board, formed to provide expert 
advice on best practice to the SD SSI staff 
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Vermont Although the NSF grant was awarded to the Vermont Department of 

Education (VDE), the Vermont SSI is governed and managed through a 
new nonprofit agency, the Vermont Institute for Science, Mathematics, 
and Technology (VISMT). VISMT was created to get broader 
participation and to avoid bureaucratic problems. However, the VDE 
also wanted SSI activities to be coordinated with statewide reform 
initiatives. Therefore, the governor appoints one member to the VISMT 
Board, the Commissioner of Education nine, and Secretary for 
Development and Community Affairs five. The VISMT Board is chaired 
by the deputy commissioner of the VDE and includes representatives 
from the schools, colleges, professional groups, and business. However, 
the majority of the Board come from business and higher education. The 
Board takes an active role in shaping policy and intends to track 
expenditures, monitor outcomes, and hold the staff accountable for 
implementing the plans that are approved. 

Table 9 presents a more general picture, classifying the statewide systemic initiatives 
by their lead agency, and suggests some of the variety in the character of SSI governance 
structures. In 1 1 states, the state education agency (SEA) is taking a lead role, although 
in 4 of these cases, the agency is sharing this responsibility with the state higher education 
authority or an institution of higher education. In three states, an institution of higher 
education serves as the key lead agency. In the other states, nonprofit organizations 
located outside the state bureaucracy and, in some cases, outside the formal education 
system, are taking the lead roles in their states' SSIs. 

We recognize that the locus of the lead agency does not necessarily indicate anything 
about the nature of its decision-making or the breadth and character of participation. Still, 
this broad range across the 21 SSI states underscores the innovative nature of the SSI 
program — and how states are often breaking the mold of traditional operating procedures 
to promote their reform agenda. Of particular interest, because of their innovative 
governance structures, are the seven states in which the SSI is officially located outside 
the state education system, and we discuss these in the next section. 




Table 9 



SSI LEAD AGENCY 



State 


State 
Education 
Agency 


SEA/State 

Higher 
Education 
Organization 


SEA/IHE 


IHE 


Preexisting 
external 
Agency 


New 
External 
Agency 


CA 


✓ 












CT 












✓ 


DE 






✓ 








FL 


✓ 












GA 








✓ 






KY 










✓ 




LA 




✓ 










MA 


✓ 












ME 












✓ 


Ml 


✓ 












MT 










✓ 




NE 








✓ 






NC 










✓ 




NM 












✓ 


OH 




✓ 










PR 








✓ 






Rl 


✓ 












SD 




✓ 










TX 


✓ 












VA 


✓ 












VT 












✓ 
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External Leadership 



Although the majority of statewide systemic initiatives are led by the state education 
agency or the state agency working in partnership with higher education, organizations 
"external" to the bureaucratic hierarchy are taking leadership roles in some states. Some 
of these, like the Kentucky Science and Technology Council and the Montana Council of 
Teachers of Mathematics, are not new, but they are taking on new responsibilities. 
Others, such as the Council for the Advancement of Mathematics and Science Education 
in New Mexico and the Connecticut Academy for Education in Science, Mathematics, and 
Technology, are new broad-based collaboratives created to promote reform. Such 
organizations are of particular interest because of their unique relationship to the formal 
decision-making hierarchy, and because of the claims that the participants make about the 
potential benefits of this approach. New or old, these "external agencies" must establish 
themselves as potent forces in state-level policy formulation. 

The seven states and their SSI lead agencies are: 



Connecticut 



Connecticut Academy for Education in Science, 
Mathematics, and Technology (new) 



Kentucky 
Maine 

New Mexico 



Kentucky Science and Technology Council 

Maine Mathematics and Science Alliance (new) 

Council for the Advancement of Mathematics and Science 
Education (new) 



Montana 



Montana Council of Teachers of Mathematics 



North Carolina 



North Carolina Science and Mathematics Alliance (a 
program of the NC Board of Science and Technology) 



Vermont 



Vermont Institute of Science, Mathematics, and 
Technology (new) 



In each of these eight cases, the state education agency is involved in the SSI. In half of 
them, the SEA is serving as the fiscal agent for the SSI. In some, the SEA is an active 
collaborator; but in all eight cases, an external group acting with some degree of 
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independence from state government and the state educational establishment is setting the 
agenda, making policy, and overseeing the implementation of the SSI. 

The compositions of these lead groups differ, but what they have in common is that 
their governing boards and their leadership are not dominated by state agencies, and that 
they operate independently of the normal state bureaucracy. A third common feature is 
that business leaders are generally playing a much larger role in guiding the SSIs than they 
do in those states where the SSIs are led by the education agencies. Below, we describe 
two of these external governance structures — a wholly new entity in Connecticut and a 
preexisting organization in Kentucky. 

Connecticut: Business-Led Collaboration in the Service of Reform 

The Connecticut Academy for Education in Science, Mathematics, and Technology 
is regarded by its advocates as the centerpiece of the state's systemic initiative and, 
potentially, the focal point for mathematics and science reform in the state. The leadership 
of the Academy is attempting to bring leaders from the schools, higher education, 
mathematics and science communities, state government, business, science-rich 
institutions, and civic organizations together to promote a common reform agenda. Led 
by a former corporate executive, the Academy staff is small but well connected to all the 
major stakeholders through the Academy's Board. Both the Board and the staff see the 
Academy as the key to the sustained and focused reform effort that they believe will be 
necessary to have a powerful impact on teaching and learning in mathematics and science. 

The Academy does not operate programs directly, but instead works through other 
institutions such as school districts, colleges, museums, the Department of Education, and 
the media. In this way, it does not compete with other institutions but seeks to strengthen 
them. The Academy sets priorities for the SSI, uses a competitive grant process to 
mobilize the resources needed for particular tasks and to get institutions working together, 
and monitors the use of funds and the success of activities. The Academy articulates the 
vision and serves as the catalyst to build public and institutional support for it. 

The uniqueness of the Academy in Connecticut's history lies in its autonomy from 
the bureaucracy and its capacity to convene key players from different sectors who have 
seldom talked in the past. The most important accomplishment in the first year was 
getting policy-makers to recognize the Academy as the lead agency in the area of 
mathematics, science, and technology. The Academy has the support of the lieutenant 
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governor, key legislators, the Department of Economic Development, the commissioner of 
higher education, and the commissioner of education. However, the degree to which the 
state Board of Education and the commissioner of education are committed to supporting 
the Academy's leadership role and to coordinating their efforts with it is not yet clear. 
There was also some resistance in the state Department of Education. Some policy- 
makers did not want an outside group conflicting with the role of the state Board of 
Education by setting policy or lobbying the legislature. Because the Academy has no legal 
authority, however, the state department officials eventually agreed to its creation and its 
role as the lead agency. 

Kentucky: Building on a Tradition of Industry-Government 
Cooperation 

Although the cooperative agreement for the SSI in Kentucky (known as PRISM) is 
between NSF and the Kentucky Department of Education (KDE), the SSI is being led and 
managed by the Kentucky Science and Technology Council (KSTC), a nonprofit 
organization, working in cooperation with the commissioner of education, who serves as 
one of the Pis. The Council was initiated by Governor Marva Collins in 1987 as an 
advisory council made up of representatives of business and academia interested in 
attracting high-technology industry to the state, improving the skills of the workforce, and 
promoting technology transfer in Kentucky. It soon become an autonomous organization. 

KSTC works with state agencies and has raised private funds from corporations and 
foundations to support its work. It serves as the state coordinator for EPSCoR 
(Experimental Program to Stimulate Competitive Research), and is working with the state 
on public-private partnerships and technology transfer. KSTC has been active in 
education for some time, running inservice programs in science and technology and 
developing curriculum modules in science. PRISM is its largest undertaking, representing 
40% of KSTC's budget. Prior to SSI, the NSF-funded elementary science project at UK 
was its main activity. 

PRISM has its own board with broad representation, including the governor, several 
key legislators, representatives from the Education and Humanities and Economic 
Development Cabinets, sev >ral SEA officials (including the commissioner), representatives 
of higher education, several business leaders, and several local officials. This board serves 
as an advisory group to KSTC but does not appear to set policy or get heavily involved in 
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management of the program. The board does provide strong links to stakeholder groups, 
and its role is still evolving. 

The president of KSTC, the KSTC SSI director, and the commissioner of education 
work closely together to coordinate the SSI with the larger reform initiative in the state. 
KSTC staff coordinate the work of the nine SSI component directors, of whom four are 
affiliated with universities, three are in KDE, and two are at KSTC itself. KSTC seems to 
have recruited the key players from the mathematics and science communities. Everyone 
with whom we spoke expressed comfort with the collegial style of decision-making used 
by KSTC staff. The component directors meet monthly to work out operational issues 
and develop plans for implementation. 

Internal Leadership 

Leadership from within existing state agencies is a much more familiar situation for 
state and federal officials who are dealing with single awards for statewide education 
improvement efforts, whether in mathematics, science, or other subjects, or for general 
school improvement efforts (such as those supported by the U.S. Department of 
Education's Chapter 2 funds under the Elementary and Secondary Education Act). As 
noted earlier in Table 9, ^bout half of the 21 SSIs have this type of leadership. Michigan, 
Texas, Rhode Island, and California are among those states in which the SSI awards have 
been made to a state agency and where the key SSI leadership is located within the state 
bureaucracy. 

Internal leadership has a variety of apparent advantages for the SSIs. For example, 
in these cases, one would expect the project to benefit from close working relationships 
with existing state and federal programs. In Michigan, the project director, located in the 
state Department of Education, was previously a coordinator of the Eisenhower 
Mathematics and Science Education Program, and thus is well positioned to understand 
the history of earlier state and local reform efforts in mathematics and science education, 
as well as to 'ake advantage of "leveraging" the Eisenhower funds for the SSI. In fact, 
Michigan reported using a sizable amount of Eisenhower funds to support its systemic 
initiative. 

On the other hand, cooperation within and across governmental agencies is not 
always easy to obtain. There are SSIs in which "turf battles" have prevented internal 
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leadership groups from obtaining the maximum benefit of their knowledge and access to 
the levers of government. 

We describe here in somewhat more detail the governance structure for one SSI in 
which the award has been made to a state agency. 

California. California has adopted an approach that keeps the systemic initiative 
closer to the key state personnel concerned with mathematics and science than have the 
SSIs governed by external agencies. Here, the principal investigators are located in the 
state Department of Education and the governor's cabinet, and the oversight responsibility 
remains largely within the state education agency. The former science supervisor for many 
years and the state's mathematics supervisor are in leadership positions, bringing with 
them a long institutional memory as well as many accomplishments in shaping the state's 
ambitious role in mathematics and science education. California's mathematics and 
science frameworks, the pressure that it placed on publishers to reform instructional 
materials, and its plans for statewide performance assessments are testimony to the work 
directed and endorsed by the state agencies over many years. 

At the same time, the California initiative does have a diverse board of directors 
external to the state department, and it has teamed up with an institution of higher 
education, the regional education laboratory, and a number of reform efforts based at 
institutions of higher education to ensure the successful implementation of its reform 
strategy. The effort thus is closely coordinated with other state reform efforts. Although 
California has paid attention to building networks with other key players in the state, 
efforts to reach out to new private-sector collaborators are not nearly on the level of some 
of the externally led SSIs, such as Connecticut's. 

Partly as a result of the decision to work within the current system, not to create a 
whole new organization nor to focus as much on coalition building at the state level, 
California has been free to focus on direct service provision. Thus, one of the key 
accomplishments of the California systemic initiative in its first year has been the 
development of two statewide teacher professional development networks reaching over 
1,000 teachers with relatively intensive professional development services (more than 30 
hours per teacher). 
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External or Internal Agencies: Finding th* Best Means of Promoting 
Reform 

Almost every 5JSI has made significant efforts 10 collaborate across several different 
agencies, and collaboration is clearly a key to effective systemic reform. It is easy to 
exaggerate the differences between governance structures that lie primarily inside or 
outside state government, and we have perhaps done so, in the interest of clarifying some 
of the strengths and weaknesses of each governance form. Nonetheless, there are some 
real differences. 

The rationale given by those who have placed the locus of decision-making for the 
reform of science, mathematics, and technology education outside of state government 
vary, but among the reasons stated are: 

• Avoidance of red tape and the capacity to act faster and to spend funds with 
fewer restrictions. 

• The ability to hire or attract better-qualified staff. 

• The capacity to build greater credibility with the mathematics and science 
communities, as well as within the business community. 

• The ability to serve as more effective advocates for reform, uninhibited by other 
state agendas or changes in state priorities. 

• The ability to sustain a long-term effort, surviving changes in state leadership. 

• The ability to make the initiative bipartisan. 

• Increased capacity to raise private funds. 

Research provides some support for the argument that nongovernmental bodies may 
have certain advantages over government as the sources and monitors standards and the 
quality of practice (Fuhrman, 1993). First, agencies external to the government may be 
able to gain greater acceptance with teachers because they are less threatening than 
government. Second, external agencies may be able to maintain membership and authority 
acror." electoral cycles more easily than government agencies, enabling them to maintain 
their focus even as political personalities and priorities change. Third, nongovernmental 
bodies may be able to attract volunteers and in-kind resources and be less costly to 
operate; they also may be better able to tap private funds so that they, and their missions, 
become less vulnerable to state budget cuts. 
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Yet, external agencies may have some serious weaknesses as well. Functioning 
outside the system, they could increase the complexity of policy-making by adding a new 
layer of consultation and review. In some cases, the creation of external agencies may 
take valuable time and resources away from substantive activities; at worst they could 
make a fragmented system less rather than more coordinated. 

The limited data we have from the evaluation's first year do not allow us to draw 
conclusions about the benefits and costs of governing a statewide systemic initiative inside 
or outside of the state bureaucracy. There are potential advantages and disadvantages to 
both. Thus for example, preliminary analysis of state initiative budgets suggests that 
externally governed initiatives are indeed more successful than their state-run counterparts 
in raising funds from external sources. Yet, not surprisingly, initiatives governed within 
the state bureaucracy appear more successful in leveraging Eisenhower and other state 
funds. 

The contrasting examples of Connecticut's and California's statewide systemic 
initiatives illustrate some of the tradeoffs involved in varying governance structures. 
Although these states are dramatically different in many ways (size, population, and 
political culture), both have enjoyed vigorous state leadership in education, reform-minded 
chief state school officers, respected capacity in the state education agencies in 
mathematics and science, cutting-edge state assessment programs, and state leadership in 
promoting new ways of teaching and learning. However, they have taken quite different 
approaches to governing and implementing their SSIs. as we have noted above. 

The differences between their two strategies may be attributed to the states' different 
political cultures or to other factors. Regardless of their origins, the important question is 
whether the governance structures in the two states and their relative focus on coalition 
building will influence the long-run success of their systemic initiatives or "buy" them 
different kinds of gains. It is, as yet, too soon to answer these questions. 

Summary and Questions 

Although it is too soon to reach conclusions about the performance of the various 
governance mechanisms, it is remarkable to note how many of the SSIs have reached 
outside existing governmental agencies for leadership and assistance. The number of cases 
in which this has occurred suggests that many states have concluded that changing the 
status quo will require creative means and new sources of support. It may not be 
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necessary that external agencies play any particular role, and it may well be that what is 
needed varies from state to slate. But it is quite possible that one contribution of the SSI 
program will be to demonstrate the contributions cf external agencies to statewide 
systemic education reform, and future findings may offer important lessons for future 
federal initiatives. 

The above discussion raises a number of questions to be addressed by the evaluation 
project, such as: 

( • } Is there any relationship between the governance structure and the capacity to 
win support for reforms from school districts, the legislature, or the public? 

(2) Is there any relationship between the governance structure and the capacity to 
raise public or private funding for mathematics and science reform? 

(3) Is there any relationship between the governance structure and the capacity of 
the initiative to be sustained past the 5 years of NSF's funding? 

(4) Do externally governed SSIs come into conflict with SEAs as a result of 
conflicts over priorities, state funding, standards, etc.? If so, how are such 
conflicts resolved? 

(5) Does the creation of new governance structures reduce the possibility of 
increasing coordination among state agencies, thus limiting the capacity to 
create more coherent state policy? 
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5 LOOKING AHEAD: CHALLENGES FOR FURTHER EVALUATION 



This report is being written after the states have had, at most, 2 years of support 
from the SSI program. Moreover, a number of the SSIs have gone through major 
reorganizations since their inception. Overall, many of the SSIs are still in either a start-up 
stage or a stage of rapid growth and evolution. Similarly, the national evaluation of the 
SSI program is at an early stage. Up to this point, all of our fieldwork has been conducted 
in only nine states, and nearly all of it has focused on people and events at the level of 
state agencies and institutions. 

The result, as we have suggested in earlier chapters, is an interim report that 
provides preliminary information. This report is largely descriptive; we have purposefully 
reached few conclusions about the SSI program or the individual SSI initiatives. Rather, 
we have concluded each chapter with specific questions for future research. Addressing 
these questions presents a number of particularly significant challenges that face the 
research team in carrying out the next phase of research. In this chapter, we focus on 
three special challenges for the evaluation: 

(1) SSI is embedded in a complex array of other reform efforts. The national 
evaluation needs to develop methods for "standing back" and examining SSI in 
this larger context. 

(2) The first year of research has focused especially on people and institutions at the 
state level. The evaluation must examine other levels of the education system 
and integrate findings across levels, from the classroom to the state capital. 

(3) The theory of systemic reform focuses on key policy issues, such as assessment 
of student learning, that are extremely complex in their own right, let alone 
when considered as parts of much larger initiatives. The evaluation needs ways 
of focusing greater attention on these complex issues while not losing sight of 
the SSI program as a whole. 

In this chapter, we focus on each of these challenges in turn and propose methods for 
meeting them. 

Understanding SSI in the Larger Context of School Reform 

A wide array of state and federal reform efforts are under way that affect schools 
and thus the SSI program itself. In the second and third chapters of the report, we 
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discussed the issues involved in assessing the statewide systemic initiatives in the context 
of ongoing state mathematics and science reforms. Yet, there are a number of other 
reforms, many of them national in scope, whose influence on SSIs we have not tracked 
closely. We discuss these below. 

School Finance Reform 

Perhaps the most dramatic and far-reaching of the school reform efforts under way 
are those resulting from court decisions in school finance cases. Typically, the stimulus for 
reform is a claim or finding that the current finance system is inequitable. Thirty-seven 
states are involved with school finance reform. In one SSI state, Kentucky, the state 
supreme court's decision to invalidate the then-prevailing method for financing schools 
(predominantly through local property taxes) was the single "cause" that led to an 
enormous number of changes, ranging from new district and school governance policies to 
brand-new, high-stakes state assessments of student achievement. Many of the 21 SSI 
states are affected, although few states have seen such rapid and sweeping impacts as 
Kentucky. Texas, Montana, and Michigan are just three of the other SSI states currently 
embroiled in revisions to their finance systems. 

School finance reforms can lead to protracted debates and political conflicts that may 
paralyze other efforts to reform education, as they did for some time in New Jersey and 
Texas. However, they also can be the stimulus for broader, more systemic reforms as in 
the case of Kentucky, and new school formulas can provide the resources needed to 
address some of the fundamental barriers to equity in mathematics and science education, 
such as the lack of qualified teachers, professional development, textbooks, lab equipment, 
computers, and other essentials in poor school districts. In states where such basic 
inequities persist, it may not be possible to close the gap in mathematics and science 
attainments. 

Other Changes in State Education Systems 

There are other potential big policy changes in the states that could affect the course 
and the impact of the SSIs. For example, the Sheff v. O Weill desegregation case in 
Connecticut has focused the state's leadership on the issue of district organization, and 
this case could undermine efforts to place mathematics and science reforms high on the 
state agenda. On the other hand, it could create an opportunity for bold initiatives to 
address the gap in access and attainment in science and mathematics for minority students. 
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The vote on vouchers in California and the property tax referendum in Colorado are 
examples of other larger educational policy decisions that might affect the course and 
progress of an SSI. 

State budget problems also are likely to affect the course of the SSIs. Reductions in 
state agency budgets reduce SEA capacity for technical assistance and slow the 
development of new assessment systems and curriculum frameworks. In spite of the 
current slow economic improvement, many states still face budget deficits and many are 
reducing their workforces. 

Federal and National Reform Efforts 

The importance of the pending reauthorization of the Chapter 1 program has already 
been noted. Chapter 1 funds turn a massive wheel whose momentum can either reinforce 
reform, impede it, or power an education system that is effectively parallel to or isolated 
from the system that SSI aims to change. 

The new administration's other education reform initiatives, Goals 2000 and the 
Schooi-to-Work Transition statutes, also could affect the SSIs. Goals 2000 calls on all 
states to develop a plan for systemic reform. This could strengthen SSIs by providing new 
impetus for the reforms in the states, strengthening public support, and generating 
additional state and federal resources; or, in the worst case, it could create competing 
coalitions and struggles over leadership and control of the reform agenda in state capitals 
and serve as a temporary distraction. The administration's School-to-Work Transition 
legislation is designed to encourage a systemic approach to education for youth leaving 
school and preparing to enter the workforce. A number of the strategies for reform 
involve program options that bring academic subjects (particularly mathematics and 
science) and vocational studies into new relationships. These strategies include the Tech.- 
Prep. model, youth apprenticeships, and the Career Academy model. Each of these 
approaches is being widely implemented and involves an applied academic curriculum that 
could well be considered consistent with the "hands-on," construct ivist theories underlying 
the NCTM Standards as well as the various efforts to develop standards for science 
education. Yet, up to this point, it appears that only a few of the SSIs have made strong 
efforts to integrate the vocational education system into their systemic reform efforts. 

Another federal program of great significance to SSI is the Eisenhower Mathematics 
and Science Education Program, administered by the U.S. Department of Education. The 




national portion of this program has recently awarded grants to 16 states to develop state 
curriculum frameworks in mathematics, science, or both. Of these 16 awards, 11 are to 
states that also have received SSI funds. The state and local portions of the Eisenhower 
program are even more critical to the SSIs. The SSI states claim that last year more than 
$20 million in Eisenhower funds was used directly in support of SSI. More than $200 
million was obligated for the state and local programs last year, and all of it was to support 
reform in mathematics and science education. Any changes in the Eisenhower program 
could have an effect on the SSI program, and the pending reauthorization of the program 
needs to be kept in mind as the national study conducts additional fieldwork this year. 

NSF itself is undertaking a new Urban Systemic Initiative. Many of the cities eligible 
for these grants are located in states with SSIs. The historic competition between state 
officials and the leadership of large cities such as New York, Los Angeles, Miami, New 
Orleans, Boston, and Detroit may require special efforts by NSF to ensure that state and 
local SSIs work together and not at cross-purposes. 

Finally, many of the SSI states are active in other national reform efforts. For 
example both the New Standards Project and Re: Learning are impacting the evolution of 
statewide reform in a number of SSI states. 

One Reform Effort, Or Many? 

All the states are involved with the Chapter 1 program, and all will be affected by 
pending federal legislation affecting Chapter 1, Eisenhower, and other programs that 
interact in one way or another with the SSL Many SSI states also have under way a 
variety of reform efforts of their own. Thus, in every SSI state, two important questions 
are how many reform efforts are being implemented and whether they are coherently 
aligned with one another. 

NSF and the U.S. Department of Education are making efforts to align their 
strategies for reform, and this attempt is important. A recent policy brief jointly developed 
by the agencies states that: 

The two agencies should produce clear signals that State activities financed 
through our respective programs should contribute to a single, integrated 
reform effort, not to one effort, funded by NSF, that focuses on mathematics 
and science education and a second one, funded by ED, that looks at the 
broader curriculum. (U.S. Department of Education; 1993, p. 3) 



State policy-makers must heed this advice, as well. Policy alignment and coherence will 
be needed for systemic reform to succeed. 



Implications for the National Evaluation 

This complex array of reform efforts has several implications for the national 
evaluation of the SSI program. First, we need to expand the list of people whom we 
interview and documents we review, especially in the case study states. For example, we 
need to interview individuals involved with the Chapter 1 program — at the state, district, 
and school levels. Of particular importance, we need to pay attention to possible tensions 
at the local level between mathematics reform and the traditional basic-skills approach on 
many Chapter 1 -funded efforts. (Typically, Chapter 1 has little direct relationship to 
science education.) 

Second, as we select two additional case study states from among the five Cohort 
Three states recently awarded SSI funding, the selection should be guided, in part, by the 
need to understand SSI in as broad a context as possible. In particular, few states in our 
current case study sample focus much effort on reform of the vocational education system, 
despite its large size and significance in the secondary education system. If we can add a 
state that is integrating SSI and the reform of vocational education, we will enlarge our 
understanding of how these reform efforts interact. 

Finally, because of the complexity of the SSIs and their contexts, our examination of 
the non-case-study states is handicapped if we cannot visit them and conduct interviews of 
a wide range of respondents. Consequently, NSF has provided the evaluation team with 
sufficient funds to make brief visits each year in a dozen of the non-case-study states. 
These will provide us with substantially more information, not only about the SSIs but 
about other reform efforts under way in these states. 

Examining the Various Levels of the Education System 

Systemic reform, by definition, involves changes throughout the educational system. 
The ambitious vision, coherence, capacity building, and mobilization of opinion in support 
of reform that have to take place on the statewide level must be matched by similar efforts 
at the local level. Ultimately, systemic reform's goal is to improve what takes place in 
classrooms between teachers and students and what students know and are able to do. 
However, up to this point in the national study, most of our attention has been devoted to 
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understanding SSI from the point of view of state-level actors. During the second and 
subsequent years of the study, we will begin to look through the educational system to 
broaden our focus to include districts, schools, classrooms, and institutions of higher 
education. 

Fieldwork this coming year (1993-94) in the nine case study states from Cohorts 
One and Two will include a significant amount of time at the local level. In all nine states, 
this will include visits to school districts, typically three per state, and to several schools in 
each of the districts (typically three per district). We plan to include both schools and 
teachers that have been directly involved in SSI-sponsored activities and schools and 
teachers that have no direct involvement. Schools in which SSI is a major factor — such as 
pilot or demonstration schools, or schools containing SSI-trained "lead teachers" — can be 
compared with others in which SSI is a less prominent factor. By studying science and 
mathematics education in comparison schools, in which SSI has not been such a significant 
factor, we hope to be better able to determine the r elative influence of SSI as compared 
with other programs and initiatives. Of course, we realize that for some elements of 
systemic reform, such as state-mandated assessment programs, it may not be possible to 
find "comparison" schools that are not affected at all. 

In addition, in those states in which SSI has a substantial focus on undergraduate 
education (including teacher preparation), local visits will include at least one institution of 
higher education. Visits to districts, schools, and institutions of higher education will be 
continued over a period of several years, so that the progress of SSI over time can be 
examined. 

At the same time, we will continue to collect information from state-level actors 
through written documents (such as annual reports to NSF) as well as interviews. This 
strategy of focusing on both the state and local levels will contribute to an understanding 
of the interaction between levels. 

Understanding Key Issues in Depth 

The first two challenges to the evaluation that we have discussed above both require 
that we be able to broaden our view of what is happening in schools to include, first, the 
influences of a wide variety of reform efforts and, second, the interaction of decisions and 
actions taken at many levels of the education system, Meeting these challenges requires 
that the evaluation have available methods that act something like a wide-angle lens, 
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taking in a very large field of view. In contrast, the third challenge requires us to narrow 
our field of view and look in much greater detail at a few, key issues. 

These issues are the ones that are linchpins of the theory of systemic reform, 
including: the creation and implementation of state curriculum frameworks in science and 
mathematics education; developing and using new student assessment systems; and how 
best to build up the capacity of the education system (e.g., by providing professional 
development to teachers with sufficient depth and breadth to make a difference). Any one 
of these issues is highly complex. We found during the first year of the evaluation that our 
methods for tracking them were insufficient. For example, we have been interested in the 
development and use of curriculum frameworks in the SSI states. We have focused on 
this issue in interviews and in our review of documents. Yet, we found that what states 
mean by "frameworks" varies so considerably that any reasonable judgments about them 
must rest on a much more thorough analysis than we carried out. We must examine the 
quality of the frameworks, their relationships to national standards, their specificity, and 
the extent of their use. 

As we move into the second year of the evaluation, we are planning more in-depth 
analyses of certain features of the SSI reform efforts, such as curriculum frameworks. We 
plan to create small teams within the core study group, charged with the task of analyzing 
certain issues across the states. We will carry out these efforts in coordination with other 
ongoing efforts of similar scope. For example, in looking at curriculum frameworks, we 
would coordinate our work with that of other NSF-funded studies (e.g., the work of the 
Council of Chief State School Officers) and with Department of Education work (e.g., the 
evaluation of the Eisenhower Framework projects). We expect that these efforts will 
result in content-specific policy briefs that will be published separately from our annual 
reports. 

We hope that these three efforts— focusing on the broader reform contexts in which 
statewide systemic initiatives are evolving; turning our attention to districts, schools, and 
classrooms; and putting effort directly into the assessment of states' efforts in key policy 
areas — will strengthen the evaluation greatly. Tracking the progress of the (now) 26 
statewide systemic initiatives and examining the strengths and weaknesses of the SSI 
program as a federal change strategy presents a daunting challenge for researchers; yet our 
task pales in comparison with the job of actually reforming the system of mathematics and 
science education to which both the states and the federal government are committed. 

71 




REFERENCES 



American Association for the Advancement of Science. (1989). Science for All 
Americans, Washington, DC: Author. 

Blank, R., & Dalkilic, M. (1992). State Policies on Science and Mathematics Education, 
1992. Washington, DC: Council of Chief State School Officers. 

California State Department of Education. (1 992). Mathematics Framework for 
California Public Schools. Sacramento, C A: Author. 

Clune, W. H. (1993). The Best Path to Systemic Educational Policy: Standard/ 

Centralized or Differentiated/Decentralized? Educational Evaluation and Policy 
Analysis, 5(15), 233-254. 

Connecticut State Board of Education. (1990). Connecticut f s Common Core of 
Learning, Hartford, CT: Connecticut State Department of Education. 

David, J. L. (1993, March). Systemic Reform: Creating the Capacity for Change 
(Draft). Prepared for the Consortium for Policy Research in Education, Rutgers 
University. Palo Alto, CA: Bay Area Research Group. 

David, J. L., & Goren, P. (1993). Transforming Education: Barriers and Expectations. 
Washington, DC: National Governors' Association. 

Elam, S. M, Rose, L. C, & Gallup, A. M. (1991). The 23rd Annual Gallup/Phi Delta 
Kappa Poll of the Public's Attitudes Toward the Public Schools. Phi Delta Kappan f 
73(1), 41-56. 

Elam, S. M, Rose, L. C, & Gallup, A. M. (1992). The 24th Annual Gallup/Phi Delta 
Kappa Poll of the Public's Attitudes Toward the Public Schools. Phi Delta Kappan t 
74(1), 41-53. 

Elmore, R. F., & McLaughlin, M. W. (1988). Steady Work: Policy, Practice, attd the 
Reform of American Education. Santa Monica, C A: RAND. 

Fuhrman, S. H. (1993). The Politics of Coherence. In S. H. Fuhrman (Ed.), Designing 
Coherent Education Policy: Improving the System. San Francisco: Jossey-Bass. 

Fuhrman, S. H., ^(une, W., & Elmore, R. (1991). Research on Education Reform: 

Lessons on the Implementation of Policy. In A. R. Odden (Ed.), Education Policy 
Implementation (pp. 197-218). Albany, NY: SUNY Press. * 



9.1 

73 



Fuhrman, S. H., & Massell, D. (1992, October). Issues and Strategies in Systemic 
Reform. New Brunswick, NJ: CPRE. 

Hawkins, E., & Zucker, A. A. (1993, September). Study of NSF's Statewide Systemic 
Initiatives (SSI) Program: Compilation of State Data from the Annual Report to 
NSF: SSI Program, 1993. Menlo Park, CA: SRI International. 

Little, J. W. (1993, Summer). Teachers' Professional Development in a Climate of 
Educational Reform. Educational Evaluation and Policy Analysis. 

Marsh, D. D., & Odden, A. R. (1991). Implementation of the California Mathematics 
and Science Curriculum Frameworks. In A. R. Odden (Ed.), Education Policy 
Implementation (pp. 219-240). Albany, NY: SUNY Press. 

Massey, W. E. (1993, February). Address to the National Science Foundation's 
Conference on Systemic Reform. 

McDonnell, L., & Elmore, R. (1987, Sumn . Getting the Job Done: Alternative Policy 
Instruments. Educational Evaluation and Policy Analysis, 6(2), 269-287. 

McLaughlin, M. W. (1987). Learning from Experience: Lessons from Policy 
Implementation. Educational Evaluation and Policy Analysis, 9, 171-178. 

McLaughlin, M. W. (1991). The Rand Change Agent Study: Ten Years Later. In A. R. 
Odden (Ed.), Education Policy Implementation (pp. 143-156). Albany, NY: 
SUNY Press. 

Meyer, M. R. (1990). Equity: The Missing Element in Recent Agendas for Mathematics 
Education. Peabody Journal of Education, 6-21. 

Michigan State Systemic Initiative. (1993). Vision Statement. 

Miles, M., & Huberman, A. M. (1984). Qualitative Data Analysis: A Sourcebook of 
New Methods. Beverly Hills, CA: Sage. 

National Council of Teachers of Mathematics. (1989). Curriculum and Evaluation 
Standards for School Mathematics. Reston, VA: Author. 

National Science Foundation. (1990). Statewide Systemic Initiatives in Science, 

Mathematics, and Engineering Education: Program Solicitation. Washington, 
DC: Author. 

NCES (National Center for Education Statistics). (1992). Digest of Education Statistics 
(Table 35). Washington, DC: Author. 

Newman, F. M, & Clune, W. H. (1992, Summer). When School Restructuring Meets 
Systemic Curriculum Reform. Brief to Policymakers, 3, 1. 




74 



O'Day, J. A., & Smith, M. S. (1993). Systemic Reform and Educational Opportunity. In 
S. Fuhrman (Ed.), Designing Coherent Education Policy: Improving the System 
(pp. 250-312). San Francisco: Jossey-Bass. 

Pechman, E. M., & Laguarda, K. G. (1993, March). Status of New State Curriculum 
Frameworks, Standards, Assessments, and Monitoring Systems. Washington, DC: 
U.S. Department of Education. 

Public Agenda Foundation. (1990). Cross Talk. New York: Author. 

Shields, P. M., Anderson, L., Bamburg, J. D., Hawkins, E. F., Knapp, M. S., Ruskus, J., 
& Wilson, C. (1993). Improving Schools from the Bottom Up: From Effective 
Schools to Restructuring. Menlo Park, CA: SRI International. 

Shields, P. M., Zucker, A. A., & Hawkins, E. (1993). Identifying Indicators forNSF's 
Statewide Systemic Initiatives (SSI) Program. Menlo Park, CA: SRI International. 

Smith, M., & O'Day, J. (199i 7 . Systemic School Reform. In S. Furhman & B. Malen 
(Eds.), The Politics of Curriculum and Testing. Bristol, PA: Falmer. 

Sutton, J., (Ed.). (1993). A Summary of Analyzed State Curriculum Frameworks. 
Aurora, CO: Midcontinent Regional Educational Laboratory. 

U.S. Department of Education, Office of Educational Research and Improvement, 
Programs for the Improvement of Practice. (1993). Building a Capacity for 
Systemic Reform (brochure). Washington, DC: Author. 



75 



STIS 




The Science & Technology Information System (STIS) 
at the National Science Foundation 



What is STIS? 

STIS is an electronic dissemination system that provides fast, 
easy access to National Science Foundation (NSF) publications. 
There is no cost to you except for possible long-distance phone 
charges. The service is available 24 hours a day, except for brief 
weekly maintenance periods. 

What Publications are Available? 

Publications currently available include: 
M The NSF Bulletin 

■ Program announcements and "Dear Colleague" letters 

■ General publications and reports 

I! Press releases, Other NSF news items 

■ NSF organizational and alphabetical phone directories 

■ NSF vacancy announcements 

■ Award abstracts (1989-now) 

Our goal is for all printed publications to be available electronically. 

Access Methods 

There are many ways to access STIS. Choose the method that 
meets your needs and the communication facilities you have 
available. 

Electronic Documents Via E-Mail If you have access to Internet 
e-mail, you can send a specially formatted message, and the docu- 
ment you request will be automatically returned to you via e-mail. 

Anonymous FTP. Internet users who are familiar with this file 
transfer method can quickly and easily transfer STIS documents 
to their local system for browsing and printing. 

On-Line STIS. If you have a VT100 emulator and an Internet 
connection or a modem, you can log on to the on-line system. The 
on-line system features full-text search and retrieval software to 
help you locate the documents and award abstracts that are of 
interest to you. Once you locate a document, you can browse 
through it on-line or download it using the Kermit protocol or 
request that it be mailed to you. 

Direct E-Mail. You can request that STIS keep you informed, via 
e-mail, of all new documents on STIS. You can elect to get either 
a summary or the full text of new documents. 

Internet Gopher and WAIS. If your campus has access to these 
Internet information resources, you can use your local client soft- 
ware to search and download NSF publications. If you have the 
capability, it is the easiest way to access STIS. 
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Getting Started with Documents Via E-Mail 

Send a message to the Internet address stisserv@nsf.gov. The text 
of the message should be as follows (the Subject line is ignored): 

get index 

You will receive a list of all the documents on STIS and instruc- 
tions for retrieving them. Please note that all requests for electron- 
ic documents should be sent to stisserv, as shown above. 
Requests for printed publications should be sent to 
pubs@nsf.gov. 

Getting Started with Anonymous FTP 

FTP to stis.nsf.gov. Enter anonymous for the usemame, and your E- 
mail address for the password. Retrieve the file "index". This con- 
tains a list of the files available on STIS and additional instructions. 

Getting Started with The On-Line System 

If you are on the Internet: telnet stis.nsf.gov. At the login 
prompt, enter public. 

If you are dialing in with a modem: Choose 1200, 2400, or 
9600 baud, 7-E-l. Dial (703) 306-0212 or (703) 306-0213 

When connected, press Enter. At the login prompt, enter public. 

Getting Started with Direct E-Mail 

Send an E-mail message to the Internet address stisserv@nsf.gov. 
Put the following in the text: 

get stisdirm 

You will receive instructions for this service. 

Getting Started with Gopher and WAIS 

The NSF Gopher server is on port 70 of stis.nsf.gov. The WAIS 
server is also on stis.nsf.gov. You can get the 4t .src" file from the 
"Directory of Servers" at quake.think.com. For further informa- 
tion contact your local computer support organization. 

For Additional Assistance Contact: 

E-mail: stis@nsf.gov (Internet) 
Phone: (703) 306-0214 (voice mail) 
TDD: (703) 306-0090 
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